Estimating Extreme 3D Image Rotations using Cascaded Attention

Shay Dekel, Yosi Keller, Martin CadíkCVPR 2024 · DOI

TL;DR

This paper introduces a novel attention-based deep learning framework for estimating extreme 3D rotations between image pairs, even when they have minimal or no overlap. Their approach leverages a cascaded attention mechanism to refine inter-image feature representations and predict rotations more accurately than existing methods, particularly in challenging scenarios.

What Problem Does This Paper Address?

Estimating the relative 3D rotation between image pairs is a fundamental problem in computer vision with applications ranging from augmented reality to autonomous driving. However, existing methods often struggle when images have limited or no overlap, a common scenario in real-world applications where viewpoints can be drastically different. Traditional approaches rely on detecting explicit geometric cues like lines or vanishing points, which can be unreliable or insufficient in such extreme cases. This paper aims to address this challenge by developing a robust deep learning method that can accurately estimate rotations even with minimal visual correspondence between image pairs.

Methods

The proposed method employs a multi-stage deep learning architecture:

  • Image Embedding Distillation (Transformer-Decoder-0): Siamese residual U-Nets with shared weights encode input images into feature maps. A weight-sharing Transformer Decoder then cross-decodes these embeddings, effectively distilling inter-image information and enhancing their representations. This step aims to create richer, more context-aware embeddings by allowing each image's representation to learn from the other.
  • Cross-Attention Computation (Transformer-Encoder): A Transformer Encoder is used to compute a cross-attention mechanism between the refined image embeddings. This replaces traditional 4D correlation volumes (4DCVs) with a more powerful attention-based approach that captures inter-image relationships. An attention mask is employed to focus solely on cross-attention, ignoring self-attention within each image's features.
  • Cascaded Attention-based Decoding (Transformer-Decoder-1 & -2): The cross-attention output is then fed into a cascade of two Transformer Decoders. The first decoder refines the cross-attention representation using a learned quaternion rotation query. The second decoder further refines this query to predict the final relative rotation. This cascaded approach allows for iterative refinement of both the attention encoding and the rotation prediction.
  • Relative Rotation Regression (MLP): A Multi-Layer Perceptron (MLP) takes the final predicted quaternion and regresses the 3D rotation. The model is trained using a regression loss on quaternions.

Datasets

The paper evaluates its approach on three benchmark datasets:

  • InteriorNet [29]: A synthetic dataset for indoor scene understanding.
  • SUN360 [54]: An indoor collection of high-resolution panoramas.
  • StreetLearn [36]: An outdoor dataset of panoramic views from urban environments.

For evaluation, the authors generated perspective images from panoramic views and created specific overlap categories (Large, Small, None) to test performance under varying degrees of overlap, mirroring the setup of prior work [7]. They also conducted cross-dataset generalization experiments using the Holicity dataset [60], training on Manhattan data and testing on London data.

Key Insights and Ideas

  • Attention for Inter-Image Relationships: The core idea is to replace traditional correlation volumes with a Transformer Encoder's cross-attention mechanism. This allows for a more sophisticated and potentially more robust way to capture relationships between features from different images, especially when overlap is minimal.
  • Cascaded Refinement: Using multiple Transformer Decoders in a cascade allows for iterative refinement of both the cross-attention representation and the rotation query. This suggests that a step-by-step refinement process can lead to more accurate predictions than a single-stage approach.
  • Distilling Information: The initial "cross-decoding" step is crucial for enhancing image embeddings by forcing them to learn from each other. This is a form of knowledge distillation between image representations.
  • Generalization to Extreme Cases: The method demonstrates strong performance on non-overlapping image pairs, a significant challenge for many existing techniques that rely on feature correspondences. This suggests the attention mechanism can infer rotation cues even without direct matches.

Key Results

The proposed method achieves state-of-the-art (SOTA) accuracy on extreme rotation estimation tasks across the evaluated datasets.

  • Overall Performance: The paper reports significant improvements over existing methods, particularly for small and non-overlapping image pairs. For instance, on the StreetLearn dataset with no overlap, the proposed method achieves a mean geodesic error of 5.33°, significantly outperforming other methods that often fail to provide estimates in such cases.
  • Cross-Dataset Generalization: The model shows superior cross-dataset generalization capabilities compared to Cai et al. [7], achieving lower average geodesic errors across different overlap categories when trained on one dataset and tested on another.
  • Qualitative Results: Visualizations (Figure 4) show accurate estimation of image footprints even for non-overlapping pairs, demonstrating the method's ability to infer rotations from subtle cues.

Limitations and Future Work

  • Computational Cost: While not explicitly detailed as a limitation, the use of Transformer Encoders and multiple decoders suggests a potentially higher computational cost compared to simpler CNN-based methods.
  • Task Specificity: The authors note that while the proposed framework is general-purpose for two-image tasks, specific modifications would be needed for other applications beyond extreme rotation estimation.
  • Future Work: The paper mentions that other applications of the attention-based architecture are reserved for future work, implying potential extensions to tasks like optical flow or image registration with different constraints. The cascaded decoding scheme could also be explored with more dual units.

Summary updated

BibTeX
@inproceedings{dekel2024estimating,
  author = {Shay Dekel and Yosi Keller and Martin Cadík},
  title = {{Estimating Extreme 3D Image Rotations using Cascaded Attention}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2024},
  doi = {10.1109/cvpr52733.2024.00250},
}
Figure