Abstract
LocalTrans is a novel multiscale network designed to solve the "cross-resolution" problem in homography estimation—aligning images with massive resolution gaps (up to 10×). By replacing memory-heavy global transformers with a specialized Local Attention Kernel (LAK) and embedding it in a multiscale architecture, the authors achieve state-of-the-art accuracy and real-time performance (60fps).
What Problem Does This Paper Address?
Homography estimation is fundamental to computer vision tasks such as image stitching and panorama generation, involving the geometric mapping between two images of the same 3D plane. However, modern "gigapixel" photography introduces a significant hurdle: cross-resolution alignment.
Imagine stitching a high-resolution, zoomed-in photo of a building's window onto a low-resolution wide-angle shot of the entire city. Traditional methods like SIFT or RANSAC struggle because local features appear fundamentally different at such disparate scales. Even recent deep learning approaches often fail as they typically "stack" images, hoping the network learns the relationship implicitly. This paper argues for explicitly modeling correspondences between different resolutions to achieve sub-pixel accuracy.
Methods
The authors introduce LocalTrans, which treats cross-resolution alignment as a multimodal problem. The architecture consists of three main pillars:
- Siamese Feature Extraction: Both the target (low-res) and unaligned (high-res) images pass through a shared-weight network to extract meaningful features.
- The Local Transformer: Standard transformers use "global attention," which is computationally expensive ($O(N^2)$). LocalTrans introduces a Local Attention Kernel (LAK). Instead of looking everywhere, it generates a unique attention map for a local neighborhood around each pixel, acting like a dynamic convolution based on image similarity.
- Multiscale Structure: The network operates in a coarse-to-fine manner across $K$ levels. By using local attention at multiple scales, the network captures short-range details at high resolutions and long-range correspondences at lower resolutions, providing "global perception" without the associated computational cost.
Datasets and Evaluation
LocalTrans was evaluated on three distinct data types:
- MS-COCO: A standard benchmark for similar-resolution settings.
- Synthesized Cross-Resolution Data: Created by downsampling MS-COCO images by 4× and 8× to simulate resolution gaps.
- Optical Zoom-in Data: Real-world data from gigapixel and cross-resolution stereo datasets, featuring resolution gaps of 6× to 10× and real-world degradations.
Key Insights
- Resolution as Modality: Viewing different resolutions as different "modalities" justifies using a Transformer—a structure successful in multimodal tasks like Visual Question Answering—to find correspondences.
- Efficiency through Locality: The LAK reduces complexity from $O(H^2 W^2)$ to $O(HW \cdot r^2)$, making the architecture viable for high-resolution processing.
- Explicit vs. Implicit Learning: Explicitly calculating cross-attention is significantly more robust than simple feature concatenation.
Key Results
LocalTrans demonstrates significant improvements in both accuracy and efficiency:
- Accuracy: On the MS-COCO 4× gap task, LocalTrans achieved a PSNR of 30.17, outperforming the previous state-of-the-art (MHN at 25.42).
- Robustness: At a 10× resolution gap on real data, LocalTrans produced artifact-free stitching, while traditional methods often failed to find enough matching points.
- Speed and Memory: The model runs at 60 FPS on an NVIDIA RTX 2080 and uses roughly 1/8th the memory of a vanilla global transformer for 256x256 images.
Limitations and Future Work
The Local Attention Kernel is currently limited to a fixed radius at each scale. While the multiscale approach helps, very large displacements might still be missed if they fall outside the kernel's range. Future work could explore the "LocalTrans" module as a drop-in component for tasks like Reference-based Super-Resolution (RefSR) or Cross-Resolution Stereo Matching.
Summary updated