LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation

Ruizhi Shao, Gaochang Wu, Yuemei Zhou, Ying Fu, Yebin Liu

Abstract

LocalTrans is a novel multiscale network designed to solve the "cross-resolution" problem in homography estimation—aligning images with massive resolution gaps (up to 10×). By replacing memory-heavy global transformers with a specialized Local Attention Kernel (LAK) and embedding it in a multiscale architecture, the authors achieve state-of-the-art accuracy and real-time performance (60fps).

What Problem Does This Paper Address?

Homography estimation is fundamental to computer vision tasks such as image stitching and panorama generation, involving the geometric mapping between two images of the same 3D plane. However, modern "gigapixel" photography introduces a significant hurdle: cross-resolution alignment.

Input images for cross-resolution homography estimation: a global low-resolution wide shot and multiple local high-resolution patches.

Imagine stitching a high-resolution, zoomed-in photo of a building's window onto a low-resolution wide-angle shot of the entire city. Traditional methods like SIFT or RANSAC struggle because local features appear fundamentally different at such disparate scales. Even recent deep learning approaches often fail as they typically "stack" images, hoping the network learns the relationship implicitly. This paper argues for explicitly modeling correspondences between different resolutions to achieve sub-pixel accuracy.

Methods

The authors introduce LocalTrans, which treats cross-resolution alignment as a multimodal problem. The architecture consists of three main pillars:

  1. Siamese Feature Extraction: Both the target (low-res) and unaligned (high-res) images pass through a shared-weight network to extract meaningful features.
  2. The Local Transformer: Standard transformers use "global attention," which is computationally expensive ($O(N^2)$). LocalTrans introduces a Local Attention Kernel (LAK). Instead of looking everywhere, it generates a unique attention map for a local neighborhood around each pixel, acting like a dynamic convolution based on image similarity.
  3. Multiscale Structure: The network operates in a coarse-to-fine manner across $K$ levels. By using local attention at multiple scales, the network captures short-range details at high resolutions and long-range correspondences at lower resolutions, providing "global perception" without the associated computational cost.

Datasets and Evaluation

LocalTrans was evaluated on three distinct data types:

  • MS-COCO: A standard benchmark for similar-resolution settings.
  • Synthesized Cross-Resolution Data: Created by downsampling MS-COCO images by 4× and 8× to simulate resolution gaps.
  • Optical Zoom-in Data: Real-world data from gigapixel and cross-resolution stereo datasets, featuring resolution gaps of 6× to 10× and real-world degradations.

Key Insights

  • Resolution as Modality: Viewing different resolutions as different "modalities" justifies using a Transformer—a structure successful in multimodal tasks like Visual Question Answering—to find correspondences.
  • Efficiency through Locality: The LAK reduces complexity from $O(H^2 W^2)$ to $O(HW \cdot r^2)$, making the architecture viable for high-resolution processing.
  • Explicit vs. Implicit Learning: Explicitly calculating cross-attention is significantly more robust than simple feature concatenation.

Key Results

LocalTrans demonstrates significant improvements in both accuracy and efficiency:

  • Accuracy: On the MS-COCO 4× gap task, LocalTrans achieved a PSNR of 30.17, outperforming the previous state-of-the-art (MHN at 25.42).
  • Robustness: At a 10× resolution gap on real data, LocalTrans produced artifact-free stitching, while traditional methods often failed to find enough matching points.
Qualitative comparison between conventional feature-based methods and LocalTrans. Note the artifacts and misalignment in the top row compared to the seamless result below.
  • Speed and Memory: The model runs at 60 FPS on an NVIDIA RTX 2080 and uses roughly 1/8th the memory of a vanilla global transformer for 256x256 images.

Limitations and Future Work

The Local Attention Kernel is currently limited to a fixed radius at each scale. While the multiscale approach helps, very large displacements might still be missed if they fall outside the kernel's range. Future work could explore the "LocalTrans" module as a drop-in component for tasks like Reference-based Super-Resolution (RefSR) or Cross-Resolution Stereo Matching.

Summary updated

Abstract

Cross-resolution image alignment is a key problem in multiscale gigapixel photography, which requires to estimate homography matrix using images with large resolution gap. Existing deep homography methods concatenate the input images or features, neglecting the explicit formulation of correspondences between them, which leads to degraded accuracy in cross-resolution challenges. In this paper, we consider the cross-resolution homography estimation as a multimodal problem, and propose a local transformer network embedded within a multiscale structure to explicitly learn correspondences between the multimodal inputs, namely, input images with different resolutions. The proposed local transformer adopts a local attention map specifically for each position in the feature. By combining the local transformer with the multiscale structure, the network is able to capture long-short range correspondences efficiently and accurately. Experiments on both the MS-COCO dataset and the real-captured cross-resolution dataset show that the proposed network outperforms existing state-of-the-art feature-based and deep-learning-based homography estimation methods, and is able to accurately align images under 10× resolution gap.
BibTeX
@inproceedings{shao2021localtrans,
  author = {Ruizhi Shao and Gaochang Wu and Yuemei Zhou and Ying Fu and Yebin Liu},
  title = {{LocalTrans: A Multiscale Local Transformer Network for Cross-Resolution Homography Estimation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2021},
  doi = {10.1109/iccv48922.2021.01462},
}
Figure