Supervised Homography Learning with Realistic Dataset Generation

Hai Jiang, Haipeng Li, Songchen Han, Haoqiang Fan, Bing Zeng, Shuaicheng Liu

Abstract

Homography estimation is a fundamental task in computer vision, yet deep learning approaches have historically been constrained by a trade-off between the stability of supervised learning on synthetic data and the realism of unsupervised learning on unlabeled video. This paper introduces RealSH, an iterative framework that bridges this gap by generating realistic, labeled training data from unlabeled real-world images. By combining dominant plane masks with sampled homographies, the framework produces a "realistic" supervised dataset (CA-sup) that allows models to outperform existing unsupervised methods in complex, dynamic scenes.

Introduction

Homography estimation—the process of calculating the geometric transformation between two images of the same scene—is essential for applications such as image stitching, video stabilization, and augmented reality. Despite its importance, training robust deep learning models for this task remains challenging due to data limitations.

Currently, researchers face two suboptimal choices:

  1. Supervised Learning: These models are numerically stable but rely on synthetic datasets (e.g., MS-COCO) created by warping static images. These datasets lack real-world complexities like parallax (depth changes) and dynamic moving objects.
  2. Unsupervised Learning: These models train on real-world video pairs without labels by minimizing photometric loss. While they encounter realistic motion, they are often unstable in challenging environments, such as low-light or low-texture scenes.

This paper addresses the "data quality" barrier by proposing a method to create datasets that possess the realism of the real world alongside the precise ground-truth labels required for supervised training.

Methodology: The RealSH Framework

The authors propose an iterative framework that alternates between data generation and model refinement. This cycle ensures that as the homography estimator improves, the quality of the generated training data also increases.

The Generation Phase (G-phase)

In this phase, the system processes unlabeled image pairs to create new, labeled training samples. The process involves:

  • Dominant Plane Detection: A network (D-Net) identifies the primary plane in a scene (e.g., a road or a wall).
  • Homography Sampling: A random "ground truth" homography is sampled.
  • Image Fusion: The system warps the source image's dominant plane to match the sampled homography and fuses it with the original target image's background.

This results in a hybrid image pair where the foreground satisfies a known mathematical homography, while the background retains the messy, realistic elements of the original video, such as moving cars or parallax artifacts.

The Training Phase (T-phase)

The homography network (H-Net) is trained on the data generated in the G-phase. To maintain high data quality, two additional modules are employed:

  • Content Consistency Module (CCM): This module uses feature-level similarity to "clean up" artifacts at the fusion boundaries, ensuring the generated images are contextually coherent and sharp.
  • Quality Assessment Module (QAM): A classifier that filters out low-quality generated pairs, rejecting transformations that appear unrealistic or failed.

Datasets and Evaluation

The study utilizes several key datasets to validate the approach:

  • MS-COCO: Used for initial synthetic pre-training.
  • CA-unsup: A large-scale unlabeled real-world dataset (800,000 pairs) used as the raw material for generation.
  • CA-sup: The authors' primary contribution—a realistic supervised dataset generated from CA-unsup.
  • GHOF: A dataset with gyroscope-guided labels used to test generalization in extreme weather conditions like rain and fog.

Key Insights

Realism vs. Label Criteria

A significant insight of this work is that a training pair does not need to be 100% governed by a single homography to be effective. By ensuring the dominant plane follows the label perfectly while allowing "non-homographic" elements (like moving objects) to persist in the background, the network learns to ignore outliers. This characteristic is vital for real-world robustness.

The Iterative Loop

By treating data generation as a learnable process, the authors bypass the need for expensive manual labeling or specialized sensors like LIDAR. The iterative nature of the framework allows the model to gradually extract more complex geometric knowledge from the base unlabeled dataset.

Results

The RealSH method achieves state-of-the-art performance across multiple benchmarks:

  • Accuracy: On the CA-unsup test set, RealSH reduced the Point Matching Error (PME) to 0.34, surpassing the previous state-of-the-art (HomoGAN at 0.39).
  • Robustness: In "Low Light" and "Low Texture" scenarios, the method reduced errors by 22.8% and 14.6%, respectively, compared to unsupervised baselines.
  • Generalization: Retraining existing supervised architectures (such as DHN or IHN) on the new CA-sup dataset led to significant performance gains, demonstrating that the quality of training data was the primary bottleneck in homography estimation.

Conclusion and Future Work

RealSH demonstrates that supervised learning can outperform unsupervised methods on real-world data if the training samples are generated with sufficient realism. While the authors note diminishing returns after approximately three iterations, the framework provides a clear path forward for geometric tasks where ground truth is difficult to obtain. Future research may extend this iterative generation strategy to other domains, such as optical flow or depth estimation.

Summary updated

Abstract

In this paper, we propose an iterative framework, which consists of two phases: a generation phase and a training phase, to generate realistic training data and yield a supervised homography network. In the generation phase, given an unlabeled image pair, we utilize the pre-estimated dominant plane masks and homography of the pair, along with another sampled homography that serves as ground truth to generate a new labeled training pair with realistic motion. In the training phase, the generated data is used to train the supervised homography network, in which the training data is refined via a content consistency module and a quality assessment module. Once an iteration is finished, the trained network is used in the next data generation phase to update the pre-estimated homography. Through such an iterative strategy, the quality of the dataset and the performance of the network can be gradually and simultaneously improved. Experimental results show that our method achieves state-of-the-art performance and existing supervised methods can be also improved based on the generated dataset. Code and dataset are available at https://github.com/JianghaiSCU/RealSH.
BibTeX
@inproceedings{jiang2023supervised,
  author = {Hai Jiang and Haipeng Li and Songchen Han and Haoqiang Fan and Bing Zeng and Shuaicheng Liu},
  title = {{Supervised Homography Learning with Realistic Dataset Generation}},
  booktitle = {IEEE/CVF International Conference on Computer Vision},
  year = {2023},
}
Figure