Abstract
The classic approach to image matching consists in the detection, description and matching of keypoints. This defines a zero-order approximation of the mapping between two images, determined by corresponding point coordinates. But the patches around keypoints typically contain more information, which may be exploited to obtain a first-order approximation of the mapping, incorporating local affine maps between corresponding keypoints. In this work, we propose a LOCal Affine Transform Estimator (LOCATE) method based on neural networks. We show that LOCATE drastically improves the accuracy of local geometry estimation by tracking inverse maps. A second contribution on guided matching and refinement is also presented. The novelty here consists in the use of LOCATE to propose new SIFT-keypoint correspondences with precise locations, orientations and scales. Our experiments show that the precision gain provided by LOCATE does play an important role in applications such as guided matching. The third contribution of this paper consists in a modification to the RANSAC algorithm, that uses LOCATE to improve the homography estimation between a pair of images. These approaches outperform RANSAC for different choices of image descriptors and image datasets, and permit to increase the probability of success in identifying image pairs in challenging matching databases.
Introduction
Traditional image matching typically relies on "zero-order" correspondences—simply finding pairs of points $(x, y)$ that look similar. However, a point is just a coordinate; it doesn't capture how the local neighborhood is stretched, rotated, or tilted when viewed from a different angle. While some descriptors (like SIFT) try to be invariant to these changes, they often fail under extreme tilts or blurs because optical blur and affine transformations do not commute.
This paper aims to bridge this gap by explicitly estimating the local affine map—a first-order Taylor approximation of the transformation—which provides a much richer description of how two images relate geometrically.
The LOCATE Network
The core contribution is the LOCATE Network, a CNN architecture designed to predict the local geometric warping between image patches. Unlike previous attempts that only predict the forward mapping, LOCATE predicts the movement of patch corners for both the direct and inverse maps (8 points total, resulting in 16 output values).
This dual-tracking approach acts as a powerful regularizer, preventing the network from "drifting" during zoom and translation—a major failure mode in prior architectures. The network takes two $60 \times 60$ patches as input and processes them through a series of convolutional layers to extract the geometric relationship.
Applications in Image Matching
The authors leverage the high-precision affine maps from LOCATE in two significant ways:
Guided Matching and Refinement
Once an initial match is found, LOCATE refines the keypoint's location, scale, and orientation. Furthermore, the estimated affine map is used to "propagate" geometry, allowing the system to search for and find new, high-precision matches in the surrounding area.
$RANSAC_{affine}$
In standard homography estimation, four point correspondences are required. However, because a local affine map provides more constraints than a single point, the authors demonstrate that a global homography can be estimated using only two matches. This reduction in the minimal sample set significantly increases the efficiency of the RANSAC algorithm. Additionally, an "affine consensus" step filters out outliers by checking if the local warp predicted by LOCATE is consistent with the global homography model.
Experimental Results
The models were trained on synthetic data from MS-COCO and evaluated on challenging real-world benchmarks including SIFT-AID, the Extreme Viewpoint Dataset (EVD), and the Oxford Affine (OxAff) dataset.
Key results include:
- Precision Boost: LOCATE outperformed state-of-the-art methods like Affnet. On the SIFT-AID dataset, it increased the average number of correct inliers nearly threefold.
- Robustness to Extreme Tilt: In the EVD dataset, where standard RANSAC often fails, the $RANSAC_{affine}$ approach achieved a high success rate.
- Lower Error: Across all datasets, the "Average Pixel Error" (AvE) was consistently lower when using LOCATE-based refinement compared to identity maps or other neural estimators.
Conclusion
LOCATE demonstrates that incorporating first-order geometric information—local affine maps—drastically improves the robustness and precision of image matching. By estimating both direct and inverse mappings, the network achieves superior stability. While the current model excels on planar surfaces, future work will focus on handling occlusions and non-planar surfaces in complex 3D scenes, as well as extending the framework to stereo matching.
Summary updated