Self-Supervised Learning of Image Scale and Orientation

Jongmin Lee, Yoonwoo Jeong, Minsu Cho

TL;DR

Researchers have developed a self-supervised framework that teaches neural networks to estimate the scale and orientation of image patches without any human-labeled data. By predicting probability distributions (histograms) and "aligning" them based on known relative transformations, their method significantly outperforms both classical algorithms like SIFT and recent deep-learning approaches in tasks like image matching and 3D reconstruction.

What Problem Does This Paper Address?

In computer vision, "local features" are the building blocks for matching images, stitching panoramas, and navigating robots. While modern Convolutional Neural Networks (CNNs) are great at handling translations (shifting an object left or right), they are notoriously sensitive to changes in scale and rotation.

To fix this, we need to assign a "characteristic pose"—a specific scale and orientation—to every interesting point in an image. If we can normalize these patches to a standard size and upright position, matching them becomes much easier. However, there’s a catch: it is incredibly difficult to create a large-scale dataset where humans have manually labeled the "correct" orientation or scale for millions of tiny, abstract image patches. Most previous methods tried to learn this implicitly through other tasks, which often led to sub-optimal results.

Methods

The authors introduce a self-supervised explicit learning framework. Instead of asking for "ground truth" labels, they use a clever geometric trick:

  1. Self-Supervised Pair Generation: They take an image patch, randomly rescale and rotate it to create a second patch, and keep track of exactly how much they changed it (the relative difference).
  2. Histogram Prediction: Instead of asking the network to output a single number (regression), which struggles with ambiguity, they design the network to output a histogram (a probability distribution) over many possible scales and orientations. This allows the model to represent multiple plausible poses (e.g., a symmetric brick might have four valid orientations).
  3. Histogram Alignment Technique: This is the core innovation. To train the model, they take the predicted histogram of the first patch, "shift" it by the known relative transformation, and then compare it to the predicted histogram of the second patch. If the model is consistent, the shifted histogram should match the second one perfectly.
  4. Architecture: They use a standard ResNet-18 backbone as a feature extractor, followed by simple multi-layer perceptrons (MLPs) to produce the final histograms.

Datasets

  • PatchPose (New): A synthetic dataset created by the authors using 1,793 images from PASCAL-VOC. It contains nearly 4 million patch pairs with known relative scale and rotation for training.
  • HPatches: A standard benchmark used to evaluate how well the model transfers to real-world viewpoint and illumination changes.
  • IMC2021 (Image Matching Challenge): Used to test the model's performance on complex 3D tasks like 6-Degree-of-Freedom (6 DoF) camera pose estimation in urban scenes.

Key Insights and Ideas

  • Histograms > Regression: Traditional regression (predicting one value) fails when a patch looks the same at 0° and 180°. By using histograms, the model can be "multi-modal," identifying several likely orientations and letting downstream processes decide which is best.
  • Relative is Enough for Absolute: The most elegant part of the paper is proving that you don't need to know the "true" orientation of a patch to learn it. As long as the model is consistent across relative changes, it naturally converges on a stable, characteristic pose for any given texture.
  • Symmetry Matters: The authors use a symmetric loss function, ensuring that the model learns the relationship from Patch A to Patch B just as well as from B to A.

Key Results

The proposed method, dubbed "ours" in the paper, shows a dramatic leap in performance:

  • Pose Accuracy: On the PatchPose test set, the model achieved 97.9% accuracy for orientation (within a 10° threshold), compared to just 28.7% for SIFT and 45.0% for the previous best learning-based method (OriNet).
  • Generalization: Even though it was trained on synthetic data, it outperformed all competitors on the real-world HPatches dataset.
  • Downstream Impact: In 6 DoF camera pose estimation (IMC2021), the method improved the number of "inlier" matches and the overall accuracy of the 3D reconstruction, proving that better local poses lead to better global maps.

Limitations and Future Work

The authors note that while the model is highly robust, its performance on the HPatches dataset is lower than on the synthetic PatchPose dataset. This is likely due to "shear" and "tilt"—geometric distortions that occur when an object is viewed from an extreme angle—which the current model (focused only on scale and rotation) does not explicitly account for.

Future work could involve extending this histogram alignment framework to handle full affine transformations (including shear and aspect ratio) to make local feature matching even more resilient to extreme perspective changes.

Summary updated

Abstract

We study the problem of learning to assign a characteristic pose, i.e., scale and orientation, for an image region of interest. Despite its apparent simplicity, the problem is non-trivial; it is hard to obtain a large-scale set of image regions with explicit pose annotations that a model directly learns from. To tackle the issue, we propose a self-supervised learning framework with a histogram alignment technique. It generates pairs of image patches by random rescaling/rotating and then train an estimator to predict their scale/orientation values so that their relative difference is consistent with the rescaling/rotating used. The estimator learns to predict a non-parametric histogram distribution of scale/orientation without any supervision. Experiments show that it significantly outperforms previous methods in scale/orientation estimation and also improves image matching and 6 DoF camera pose estimation by incorporating our patch poses into a matching process.
BibTeX
@inproceedings{lee2021self,
  author = {Jongmin Lee and Yoonwoo Jeong and Minsu Cho},
  title = {{Self-Supervised Learning of Image Scale and Orientation}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=PcqcFcvr7},
}

Loading related content...

Figure