Object landmark discovery through unsupervised adaptation

Enrique Sanchez, Georgios Tzimiropoulos

Abstract

This paper proposes a method to ease the unsupervised learning of object landmark detectors. Similarly to previous methods, our approach is fully unsupervised in a sense that it does not require or make any use of annotated landmarks for the target object category. Contrary to previous works, we do however assume that a landmark detector, which has already learned a structured representation for a given object category in a fully supervised manner, is available. Under this setting, our main idea boils down to adapting the given pre-trained network to the target object categories in a fully unsupervised manner. To this end, our method uses the pre-trained network as a core which remains frozen and does not get updated during training, and learns, in an unsupervised manner, only a projection matrix to perform the adaptation to the target categories. By building upon an existing structured representation learned in a supervised manner, the optimization problem solved by our method is much more constrained with significantly less parameters to learn which seems to be important for the case of unsupervised learning. We show that our method surpasses fully unsupervised techniques trained from scratch as well as a strong baseline based on fine-tuning, and produces state-of-the-art results on several datasets.

Introduction

Detecting landmarks is a cornerstone of computer vision, essential for tasks like face swapping, gesture recognition, and 3D reconstruction. Traditionally, this requires thousands of images manually annotated by humans—a tedious and expensive process. While "unsupervised" methods exist to find these points without labels, they typically struggle with four main challenges: ambiguity in defining a "good" landmark, learning semantic parts, maintaining consistency under viewpoint changes, and handling object deformation.

Most unsupervised models try to learn these features from scratch, which often leads to "unstable" landmarks that jitter or disappear when the image changes slightly. This paper introduces a novel way to discover object landmarks by leveraging existing structured knowledge from a different domain.

Methodology

The authors propose a "Domain Adaptation" approach. Instead of starting with a blank slate, they utilize a Core Network (a Stacked Hourglass architecture) already trained via supervised learning on a well-documented category: human pose estimation (MPII dataset).

The "Frozen Core" Strategy

The key innovation is that the weights of this core network remain frozen. To adapt the network to a new category (e.g., cats or shoes), the model only learns a projection matrix for each convolutional layer. This matrix transforms the "knowledge" of human structure into the structure of the target object.

Conceptual comparison of supervised, unsupervised, and the proposed adaptation approach.

The Training Loop

To train this without labels, the method uses an auxiliary task: Conditional Image Generation.

  1. The model takes an image and predicts landmarks.
  2. A separate "Generator" network tries to reconstruct the original image using only those landmarks and a deformed version of the image.
  3. If the landmarks are stable and semantically meaningful, the generator can successfully reconstruct the object.

By only learning the projection matrices, the optimization problem is much more constrained, preventing the model from falling into the "noisy" solutions that plague from-scratch unsupervised learning.

Experimental Setup

The researchers evaluated their method across several diverse datasets:

  • Source Domain: MPII (Human Pose) used for the pre-trained core.
  • Target Domains: CelebA (Faces), UT-Zappos50k (Shoes), and the Cats Head dataset.
  • Evaluation Benchmarks: MAFL and AFLW (Face landmarks), LS3D (Large-scale 3D faces), and BBC-Pose (Human body).

Results and Discussion

The proposed method achieves state-of-the-art stability and accuracy while training only about 10% of the usual parameters (~0.5M vs ~6M in a standard Hourglass model).

Qualitative comparison of landmark discovery on faces and cats across different training strategies.

Key Insights

  • Structure is Universal: A network trained to find human elbows and knees already possesses a "structured representation" of geometry that is useful for finding the ears of a cat or the heel of a boot.
  • Efficiency and Regularization: By learning only the projection matrices, the model is forced to be more efficient. This constraint acts as a powerful regularizer, leading to much higher landmark stability than standard fine-tuning.
  • Quantitative Accuracy: On the MAFL dataset, the method achieved an error rate of 3.99, outperforming several complex generative and equivariance-based unsupervised baselines.

Conclusion and Future Work

While the method surpasses fully unsupervised techniques trained from scratch, the authors noted a performance gap on the LS3D dataset, suggesting that unsupervised discovery is still very challenging when objects undergo extreme 3D rotations. Future work may explore "few-shot" learning, as the small number of parameters makes this a prime candidate for training with only a handful of labeled examples. Additionally, the method's success relies on having a strong core network; the choice of the initial supervised task remains a critical factor in the quality of the resulting adaptation.

Summary updated

Abstract

This paper proposes a method to ease the unsupervised learning of object landmark detectors. Similarly to previous methods, our approach is fully unsupervised in a sense that it does not require or make any use of annotated landmarks for the target object category. Contrary to previous works, we do however assume that a landmark detector, which has already learned a structured representation for a given object category in a fully supervised manner, is available. Under this setting, our main idea boils down to adapting the given pre-trained network to the target object categories in a fully unsupervised manner. To this end, our method uses the pre-trained network as a core which remains frozen and does not get updated during training, and learns, in an unsupervised manner, only a projection matrix to perform the adaptation to the target categories. By building upon an existing structured representation learned in a supervised manner, the optimization problem solved by our method is much more constrained with significantly less parameters to learn which seems to be important for the case of unsupervised learning. We show that our method surpasses fully unsupervised techniques trained from scratch as well as a strong baseline based on fine-tuning, and produces state-of-the-art results on several datasets. Code can be found at https://github.com/ESanchezLozano/SAIC-Unsupervised-landmark-detection-NeurIPS2019 .
BibTeX
@inproceedings{sanchez2019object,
  author = {Enrique Sanchez and Georgios Tzimiropoulos},
  title = {{Object landmark discovery through unsupervised adaptation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=WRBzHQe40lI},
}
Figure