Introduction
Object landmark detection—identifying specific semantic points like the corners of eyes or knee joints—is a cornerstone of computer vision. Traditionally, training models for this task requires massive datasets with thousands of manual annotations, a labor-intensive process that is impractical to scale across every possible object category.
Existing unsupervised methods typically rely on two principles: 2D equivariance (the idea that landmarks should move predictably when an image is transformed) or image generation (using landmarks as an intermediate bottleneck to reconstruct an image). However, these methods often struggle with "out-of-plane" rotations, such as a face turning from a frontal view to a profile view. Consequently, they frequently discover landmarks that lack semantic meaning to humans or lose track of points entirely during extreme 3D viewpoint changes.
This paper presents a novel unsupervised framework that bridges this gap by focusing on correspondence: ensuring that a discovered landmark represents the same physical part of an object across different images and viewpoints. By combining self-training with cross-image correspondence, the approach successfully captures semantically meaningful landmarks even under extreme 3D viewpoint changes where previous generative or equivariant models fail.
The Self-Training and Correspondence Pipeline
The authors propose a two-stage pipeline that moves away from generative proxies and focuses directly on the geometry of the object.
Stage 1: Recovering Correspondence & Self-Training
The process begins with a generic, noisy keypoint detector (SuperPoint) to provide an initial population of points. This leverages a "noise-first" learning property: neural networks tend to learn consistent, "clean" patterns before they overfit to noise.
A shared backbone feeds into a dual-head architecture:
- Detector Head: Learns to find any stable landmark.
- Feature Extractor Head: Generates descriptors for those detected points.
To find correspondences, the model clusters these descriptors across the entire dataset. To ensure a single image doesn't assign multiple points to the same landmark, the authors use the Hungarian algorithm to enforce a one-to-one mapping between detected points and cluster centroids.
Through self-training, the detector improves itself by using its own high-confidence outputs from previous rounds as "pseudo-ground truth." This naturally filters out the erratic initial keypoints and hones in on stable object parts.
Stage 2: Learning the Final Detector
In the second stage, the model trains a final landmark detector to regress $M$ heatmaps. The method utilizes an over-segmentation strategy for viewpoint invariance. Initially, the model uses a large number of clusters ($M \gg K$) to capture different views of the same part (e.g., the "left side of the nose" in a profile view versus a frontal view). During this stage, clusters that consistently fire at the same spatial locations are progressively merged into a final, concise set of $K$ semantic landmarks.
Key Results and Performance
The researchers validated their method across several diverse and challenging datasets, including faces (CelebA, AFLW, LS3D), human bodies (BBCPose, Human3.6M), and animals (Cat Heads). The method achieved state-of-the-art results in unsupervised landmark discovery, particularly excelling on datasets with high pose variability.
- Handling Extreme Poses (LS3D): Qualitatively, the model maintained landmark correspondence as faces rotated into full profile views. Previous state-of-the-art methods (like Jakab et al. or Zhang et al.) typically saw their point clouds collapse or lose track of features during these out-of-plane rotations.
- Human Body Landmarks (Human3.6M): Tested without temporal or video information to evaluate static discovery, the model achieved an average accuracy of 70.35% (measured by points within a 12px radius). This vastly outperforms previous unsupervised methods, which historically hovered between 23% and 42%.
- Semantic Alignment: Despite being entirely unsupervised, the discovered landmarks aligned closely with human-annotated points (like eyes, nose, and joints), suggesting the model successfully captured the underlying 3D structure of the objects.
Limitations and Future Work
While highly effective, the approach has a few notable limitations. First, performance is heavily reliant on the initial keypoint population. If the initial generic detector (SuperPoint) fails to pick up a specific part—such as a feature on a low-texture surface—the model is unlikely to "discover" it later. Second, while the results look semantically meaningful, the method does not mathematically guarantee that discovered points will perfectly correspond to human-defined parts.
Finally, while significantly improved over prior work, the discovered landmarks are not yet perfectly invariant to all 3D rotations. The authors suggest that enforcing stricter consistency across viewpoints is a promising direction for future research in unsupervised landmark discovery.
Summary updated