Self-Supervised Learning of Image Scale and Orientation

Jongmin Lee, Yoonwoo Jeong, Minsu Cho

Abstract

We study the problem of learning to assign a characteristic pose, i.e., scale and orientation, for an image region of interest. Despite its apparent simplicity, the problem is non-trivial; it is hard to obtain a large-scale set of image regions with explicit pose annotations that a model directly learns from. To tackle the issue, we propose a self-supervised learning framework with a histogram alignment technique. It generates pairs of image patches by random rescaling/rotating and then train an estimator to predict their scale/orientation values so that their relative difference is consistent with the rescaling/rotating used. The estimator learns to predict a non-parametric histogram distribution of scale/orientation without any supervision. Experiments show that it significantly outperforms previous methods in scale/orientation estimation and also improves image matching and 6 DoF camera pose estimation by incorporating our patch poses into a matching process.

Introduction

In computer vision, local features serve as the fundamental building blocks for tasks such as image matching, panorama stitching, and robotic navigation. While modern Convolutional Neural Networks (CNNs) exhibit high proficiency in handling translations, they remain sensitive to variations in scale and rotation. To achieve robustness, it is necessary to assign a "characteristic pose"—a specific scale and orientation—to every region of interest. Normalizing these patches to a standard size and upright position significantly simplifies the matching process.

However, obtaining ground-truth labels for the "correct" orientation or scale of millions of abstract image patches is practically impossible. Previous attempts to learn these parameters implicitly through auxiliary tasks often resulted in sub-optimal performance. This work addresses the challenge by proposing a framework that learns these poses explicitly without human supervision.

Methodology

The core of the proposed approach is a self-supervised explicit learning framework that leverages geometric consistency.

Self-Supervised Framework

The training process begins with the generation of image patch pairs. Given an initial patch $I$, a second patch $I'$ is created through random rescaling and rotation. The exact relative transformation (scale change $\Delta_s$ and orientation change $\Delta_o$) is recorded and used as the supervisory signal.

The proposed self-supervised framework uses PatchScaNet and PatchOriNet to predict histograms, which are then aligned using known relative transformations.

Histogram Prediction and Alignment

Rather than using standard regression to predict a single scalar value—which often fails due to visual ambiguities or symmetries (e.g., a brick looking identical at 0° and 180°)—the network is designed to output a non-parametric histogram distribution. This allows the model to represent multiple plausible poses simultaneously.

The "histogram alignment" technique is the key innovation. The model predicts histograms $h$ and $h'$ for patches $I$ and $I'$, respectively. During training, $h$ is shifted by the known relative transformation $\Delta$. The loss function then encourages the shifted version of $h$ to match $h'$. For orientation, a cyclic-shift is employed to account for the periodic nature of angles. This consistency requirement forces the model to converge on a stable, characteristic pose for any given texture.

Experimental Results

The model was trained on the new PatchPose dataset, which contains nearly 4 million patch pairs derived from PASCAL-VOC images. Evaluation was conducted on standard benchmarks like HPatches and the IMC2021 (Image Matching Challenge).

Pose Accuracy

On the PatchPose test set, the proposed method achieved a remarkable 97.9% accuracy for orientation (within a 10° threshold), vastly outperforming classical SIFT (28.7%) and the previous state-of-the-art learning-based method, OriNet (45.0%).

Qualitative comparison showing the proposed method's superior accuracy in predicting scale and orientation compared to SIFT, relative to ground truth (GT).

Downstream Performance

The benefits of improved pose estimation extend to complex 3D tasks. In 6-Degree-of-Freedom (6 DoF) camera pose estimation, integrating the proposed patch poses led to a higher number of "inlier" matches and improved overall 3D reconstruction accuracy. This demonstrates that more accurate local poses directly contribute to more robust global mapping.

Conclusion

This research demonstrates that characteristic image poses can be learned effectively through self-supervision and histogram alignment. By moving away from scalar regression and toward distribution-based consistency, the model achieves unprecedented accuracy in scale and orientation estimation. While the current model is highly robust, future work could extend this framework to handle full affine transformations, including shear and tilt, to further enhance resilience against extreme perspective changes.

Summary updated

Abstract

We study the problem of learning to assign a characteristic pose, i.e., scale and orientation, for an image region of interest. Despite its apparent simplicity, the problem is non-trivial; it is hard to obtain a large-scale set of image regions with explicit pose annotations that a model directly learns from. To tackle the issue, we propose a self-supervised learning framework with a histogram alignment technique. It generates pairs of image patches by random rescaling/rotating and then train an estimator to predict their scale/orientation values so that their relative difference is consistent with the rescaling/rotating used. The estimator learns to predict a non-parametric histogram distribution of scale/orientation without any supervision. Experiments show that it significantly outperforms previous methods in scale/orientation estimation and also improves image matching and 6 DoF camera pose estimation by incorporating our patch poses into a matching process.
BibTeX
@inproceedings{lee2021self,
  author = {Jongmin Lee and Yoonwoo Jeong and Minsu Cho},
  title = {{Self-Supervised Learning of Image Scale and Orientation}},
  booktitle = {BMVC},
  year = {2021},
  url = {https://openreview.net/forum?id=PcqcFcvr7},
}
Figure