Abstract
Homography estimation is a fundamental task in computer vision, yet deep learning approaches have historically been constrained by a trade-off between the stability of supervised learning on synthetic data and the realism of unsupervised learning on unlabeled video. This paper introduces RealSH, an iterative framework that bridges this gap by generating realistic, labeled training data from unlabeled real-world images. By combining dominant plane masks with sampled homographies, the framework produces a "realistic" supervised dataset (CA-sup) that allows models to outperform existing unsupervised methods in complex, dynamic scenes.
Introduction
Homography estimation—the process of calculating the geometric transformation between two images of the same scene—is essential for applications such as image stitching, video stabilization, and augmented reality. Despite its importance, training robust deep learning models for this task remains challenging due to data limitations.
Currently, researchers face two suboptimal choices:
- Supervised Learning: These models are numerically stable but rely on synthetic datasets (e.g., MS-COCO) created by warping static images. These datasets lack real-world complexities like parallax (depth changes) and dynamic moving objects.
- Unsupervised Learning: These models train on real-world video pairs without labels by minimizing photometric loss. While they encounter realistic motion, they are often unstable in challenging environments, such as low-light or low-texture scenes.
This paper addresses the "data quality" barrier by proposing a method to create datasets that possess the realism of the real world alongside the precise ground-truth labels required for supervised training.
Methodology: The RealSH Framework
The authors propose an iterative framework that alternates between data generation and model refinement. This cycle ensures that as the homography estimator improves, the quality of the generated training data also increases.
The Generation Phase (G-phase)
In this phase, the system processes unlabeled image pairs to create new, labeled training samples. The process involves:
- Dominant Plane Detection: A network (D-Net) identifies the primary plane in a scene (e.g., a road or a wall).
- Homography Sampling: A random "ground truth" homography is sampled.
- Image Fusion: The system warps the source image's dominant plane to match the sampled homography and fuses it with the original target image's background.
This results in a hybrid image pair where the foreground satisfies a known mathematical homography, while the background retains the messy, realistic elements of the original video, such as moving cars or parallax artifacts.
The Training Phase (T-phase)
The homography network (H-Net) is trained on the data generated in the G-phase. To maintain high data quality, two additional modules are employed:
- Content Consistency Module (CCM): This module uses feature-level similarity to "clean up" artifacts at the fusion boundaries, ensuring the generated images are contextually coherent and sharp.
- Quality Assessment Module (QAM): A classifier that filters out low-quality generated pairs, rejecting transformations that appear unrealistic or failed.
Datasets and Evaluation
The study utilizes several key datasets to validate the approach:
- MS-COCO: Used for initial synthetic pre-training.
- CA-unsup: A large-scale unlabeled real-world dataset (800,000 pairs) used as the raw material for generation.
- CA-sup: The authors' primary contribution—a realistic supervised dataset generated from CA-unsup.
- GHOF: A dataset with gyroscope-guided labels used to test generalization in extreme weather conditions like rain and fog.
Key Insights
Realism vs. Label Criteria
A significant insight of this work is that a training pair does not need to be 100% governed by a single homography to be effective. By ensuring the dominant plane follows the label perfectly while allowing "non-homographic" elements (like moving objects) to persist in the background, the network learns to ignore outliers. This characteristic is vital for real-world robustness.
The Iterative Loop
By treating data generation as a learnable process, the authors bypass the need for expensive manual labeling or specialized sensors like LIDAR. The iterative nature of the framework allows the model to gradually extract more complex geometric knowledge from the base unlabeled dataset.
Results
The RealSH method achieves state-of-the-art performance across multiple benchmarks:
- Accuracy: On the CA-unsup test set, RealSH reduced the Point Matching Error (PME) to 0.34, surpassing the previous state-of-the-art (HomoGAN at 0.39).
- Robustness: In "Low Light" and "Low Texture" scenarios, the method reduced errors by 22.8% and 14.6%, respectively, compared to unsupervised baselines.
- Generalization: Retraining existing supervised architectures (such as DHN or IHN) on the new CA-sup dataset led to significant performance gains, demonstrating that the quality of training data was the primary bottleneck in homography estimation.
Conclusion and Future Work
RealSH demonstrates that supervised learning can outperform unsupervised methods on real-world data if the training samples are generated with sufficient realism. While the authors note diminishing returns after approximately three iterations, the framework provides a clear path forward for geometric tasks where ground truth is difficult to obtain. Future research may extend this iterative generation strategy to other domains, such as optical flow or depth estimation.
Summary updated