TL;DR
Street view image matching is crucial for Augmented Reality (AR), but illumination changes severely degrade accuracy. This paper introduces a novel feature descriptor by combining Center-Symmetric Local Binary Patterns (CS-LBP) with existing descriptors. Their experiments show this combined approach significantly improves matching accuracy under varying illumination, making AR systems more robust.
What Problem Does This Paper Address?
Augmented Reality (AR) systems rely heavily on accurately overlaying virtual content onto the real world. A key component of this is markerless AR, which uses feature matching to understand the camera's position and orientation relative to the environment. Street view images, commonly used in AR, are particularly susceptible to drastic changes in illumination due to factors like time of day, weather, and shadows. These illumination variations cause the same scene to appear very different across images, making traditional feature matching algorithms struggle to find corresponding points, leading to inaccurate AR experiences. Existing algorithms like SIFT, SURF, and ORB, while powerful, are often sensitive to these lighting changes, limiting their effectiveness in real-world AR applications.
Methods
The core innovation of this paper is the development of combined feature descriptors. They start with eight well-established feature descriptors: SIFT, SURF, DAISY, BRISK, ORB, FREAK, KAZE, and AKAZE. For each of these, they create a new, enhanced descriptor by concatenating the original descriptor with the Center-Symmetric Local Binary Pattern (CS-LBP) descriptor.
CS-LBP is a descriptor known for its robustness to illumination changes. The paper details the steps for computing CS-LBP:
- Keypoint Identification: A keypoint is detected with its coordinate, scale, and orientation.
- Region Extraction and Rotation: A 9x9 neighborhood around the keypoint is extracted and rotated to align with the keypoint's orientation, ensuring rotation invariance.
- CS-LBP Computation: For each pixel in the rotated neighborhood, a binary value is computed based on the intensity difference between symmetric pixel pairs relative to a threshold. This results in an 81-dimensional binary pattern.
- Descriptor Combination:
- For binary original descriptors (BRISK, ORB, FREAK, AKAZE), the 81-dimensional CS-LBP is directly concatenated.
- For non-binary original descriptors (SIFT, SURF, DAISY, KAZE), the CS-LBP vector is normalized before concatenation.
Finally, these combined descriptors are used for matching keypoints between images using the nearest distance ratio method, employing Euclidean distance for non-binary descriptors and Hamming distance for binary ones.
Datasets
The experiments were conducted on five benchmark street view image sequences provided by Mikolajczyk, commonly used for evaluating image matching algorithms:
- Leuven sequence: Features only illumination changes.
- Bikes sequence: Features blur and illumination changes.
- Boat sequence: Features scale, rotation, and illumination changes.
- Graffiti sequence: Features viewpoint and illumination changes.
- Ubc sequence: Features JPEG compression and illumination changes.
These sequences were chosen to represent a variety of challenging conditions that AR systems might encounter.
Key Insights and Ideas
- Illumination is a Major Hurdle for AR: The paper strongly emphasizes that illumination variations are a primary cause of failure for feature matching in AR, directly impacting system accuracy.
- CS-LBP as a Robust Building Block: The CS-LBP descriptor, while simple, proves to be a valuable component for enhancing illumination invariance.
- Concatenation is Effective: Simply combining existing descriptors with CS-LBP significantly boosts performance, suggesting that complementary information is captured.
- Different Descriptors Excel in Different Conditions: The "best" combined descriptor varies depending on the specific challenges (illumination, blur, scale, viewpoint, compression), highlighting the need for adaptive approaches or careful selection. For instance, FREAK-CSLBP was best for pure illumination, while SURF-CSLBP was good for blur and illumination.
Key Results
The experimental results demonstrate a clear improvement in matching accuracy for the combined descriptors across various challenging scenarios:
- Illumination Changes (Leuven sequence): Combined descriptors consistently showed higher precision than original descriptors, with FREAK-CSLBP achieving over 90% precision. Recall was slightly reduced but acceptable.
- Blur and Illumination Changes (Bikes sequence): SURF-CSLBP showed the best performance, maintaining high precision and a minimal recall reduction.
- Scale, Rotation, and Illumination Changes (Boat sequence): ORB-CSLBP performed best, achieving high precision and good recall.
- Viewpoint and Illumination Changes (Graffiti sequence): FREAK-CSLBP again showed strong performance with high precision.
- JPEG Compression and Illumination Changes (Ubc sequence): ORB-CSLBP demonstrated the best matching performance.
In terms of speed, the CS-LBP descriptor itself is computationally efficient. The combined descriptors generally incurred a slight increase in computational and matching time compared to their original counterparts, but this increase was deemed acceptable for real-time AR applications.
Limitations and Future Work
- Recall Reduction: A notable limitation is that while precision generally increased, the recall (number of correctly matched keypoints) sometimes decreased. The authors attribute this to a stricter filtering process when combining descriptors, potentially leading to the excessive discarding of valid keypoints.
- Future Work: The authors plan to further improve the combined descriptor to address this over-filtering problem. They also aim to enhance the computational efficiency of these algorithms to further boost the speed of real-time AR systems. The paper suggests that with advancements in technologies like 5G, AR systems will see rapid improvements, and robust feature matching will be critical.
Summary updated