Unsupervised Learning of Object Landmarks via Self-Training Correspondence

Dimitrios Mallis, Enrique Sanchez, Matthew Bell, Georgios TzimiropoulosNeurIPS 2020 · OpenReview

Introduction

Object landmark detection—identifying specific semantic points like the corners of eyes or knee joints—is a cornerstone of computer vision. Traditionally, training models for this task requires massive datasets with thousands of manual annotations, a labor-intensive process that is impractical to scale across every possible object category.

Existing unsupervised methods typically rely on two principles: 2D equivariance (the idea that landmarks should move predictably when an image is transformed) or image generation (using landmarks as an intermediate bottleneck to reconstruct an image). However, these methods often struggle with "out-of-plane" rotations, such as a face turning from a frontal view to a profile view. Consequently, they frequently discover landmarks that lack semantic meaning to humans or lose track of points entirely during extreme 3D viewpoint changes.

This paper presents a novel unsupervised framework that bridges this gap by focusing on correspondence: ensuring that a discovered landmark represents the same physical part of an object across different images and viewpoints. By combining self-training with cross-image correspondence, the approach successfully captures semantically meaningful landmarks even under extreme 3D viewpoint changes where previous generative or equivariant models fail.

The Self-Training and Correspondence Pipeline

The authors propose a two-stage pipeline that moves away from generative proxies and focuses directly on the geometry of the object.

Stage 1: Recovering Correspondence & Self-Training

The process begins with a generic, noisy keypoint detector (SuperPoint) to provide an initial population of points. This leverages a "noise-first" learning property: neural networks tend to learn consistent, "clean" patterns before they overfit to noise.

A shared backbone feeds into a dual-head architecture:

  1. Detector Head: Learns to find any stable landmark.
  2. Feature Extractor Head: Generates descriptors for those detected points.

To find correspondences, the model clusters these descriptors across the entire dataset. To ensure a single image doesn't assign multiple points to the same landmark, the authors use the Hungarian algorithm to enforce a one-to-one mapping between detected points and cluster centroids.

t-SNE visualization comparing initial SuperPoint features to the learned, clustered features.

Through self-training, the detector improves itself by using its own high-confidence outputs from previous rounds as "pseudo-ground truth." This naturally filters out the erratic initial keypoints and hones in on stable object parts.

Stage 2: Learning the Final Detector

In the second stage, the model trains a final landmark detector to regress $M$ heatmaps. The method utilizes an over-segmentation strategy for viewpoint invariance. Initially, the model uses a large number of clusters ($M \gg K$) to capture different views of the same part (e.g., the "left side of the nose" in a profile view versus a frontal view). During this stage, clusters that consistently fire at the same spatial locations are progressively merged into a final, concise set of $K$ semantic landmarks.

Key Results and Performance

The researchers validated their method across several diverse and challenging datasets, including faces (CelebA, AFLW, LS3D), human bodies (BBCPose, Human3.6M), and animals (Cat Heads). The method achieved state-of-the-art results in unsupervised landmark discovery, particularly excelling on datasets with high pose variability.

  • Handling Extreme Poses (LS3D): Qualitatively, the model maintained landmark correspondence as faces rotated into full profile views. Previous state-of-the-art methods (like Jakab et al. or Zhang et al.) typically saw their point clouds collapse or lose track of features during these out-of-plane rotations.
Qualitative comparison of landmark detection across extreme yaw angles.
  • Human Body Landmarks (Human3.6M): Tested without temporal or video information to evaluate static discovery, the model achieved an average accuracy of 70.35% (measured by points within a 12px radius). This vastly outperforms previous unsupervised methods, which historically hovered between 23% and 42%.
  • Semantic Alignment: Despite being entirely unsupervised, the discovered landmarks aligned closely with human-annotated points (like eyes, nose, and joints), suggesting the model successfully captured the underlying 3D structure of the objects.

Limitations and Future Work

While highly effective, the approach has a few notable limitations. First, performance is heavily reliant on the initial keypoint population. If the initial generic detector (SuperPoint) fails to pick up a specific part—such as a feature on a low-texture surface—the model is unlikely to "discover" it later. Second, while the results look semantically meaningful, the method does not mathematically guarantee that discovered points will perfectly correspond to human-defined parts.

Finally, while significantly improved over prior work, the discovered landmarks are not yet perfectly invariant to all 3D rotations. The authors suggest that enforcing stricter consistency across viewpoints is a promising direction for future research in unsupervised landmark discovery.

Summary updated

Abstract

This paper addresses the problem of unsupervised discovery of object landmarks. We take a different path compared to that of existing works, based on 2 novel perspectives: (1) Self-training: starting from generic keypoints, we propose a self-training approach where the goal is to learn a detector that improves itself becoming more and more tuned to object landmarks. (2) Correspondence: we identify correspondence as a key objective for unsupervised landmark discovery and propose an optimization scheme which alternates between recovering object landmark correspondence across different images via clustering and learning an object landmark descriptor without labels. Compared to previous works, our approach can learn landmarks that are more flexible in terms of capturing large changes in viewpoint. We show the favourable properties of our method on a variety of difficult datasets including LS3D, BBCPose and Human3.6M. Code is available at https://github.com/malldimi1/UnsupervisedLandmarks
BibTeX
@inproceedings{mallis2020unsupervised,
  author = {Dimitrios Mallis and Enrique Sanchez and Matthew Bell and Georgios Tzimiropoulos},
  title = {{Unsupervised Learning of Object Landmarks via Self-Training Correspondence}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2020},
  url = {https://openreview.net/forum?id=p4UXLiqRAFt},
}
Figure