Object landmark discovery through unsupervised adaptation

Enrique Sanchez, Georgios Tzimiropoulos

Introduction

This paper introduces a novel approach to discovering object landmarks—such as the corners of eyes, the tip of a shoe, or the ears of a cat—without relying on manual labels. By taking a model already trained to find landmarks on a source category (e.g., human pose) and "nudging" it to adapt to a new target category using an unsupervised reconstruction task, the authors achieve state-of-the-art stability and accuracy with significantly less training effort.

The Challenge of Landmark Detection

Landmark detection involves identifying specific semantic points on an object. It is a cornerstone of computer vision, essential for tasks like face swapping, gesture recognition, and 3D reconstruction. Traditionally, this requires "supervised learning," where humans painstakingly click on thousands of images to mark keypoints. This process is slow, expensive, and scales poorly to the vast number of object categories in existence.

"Unsupervised" landmark discovery aims to find these points automatically. However, learning from scratch is incredibly difficult because landmarks are inherently ambiguous. Without guidance, a model might pick a random patch of texture instead of a meaningful anatomical part. Previous unsupervised methods often produced "unstable" landmarks that jittered or disappeared when the object moved or rotated.

Methodology: Unsupervised Adaptation

The authors propose a Domain Adaptation approach. Their core hypothesis is that a model that has already learned the "concept" of landmarks for one category has developed a structured internal representation that can be reused for others.

Comparison of network parameter updates: supervised, unsupervised, and the proposed hybrid approach.
  1. The Core Network: The system starts with a "core" network (an Hourglass architecture) pre-trained on a supervised task, such as human pose estimation.
  2. Frozen Knowledge: During the unsupervised phase, the weights of this core network are frozen. It is not allowed to change what it knows about low-level features and structures.
  3. The Adaptation Layer: To adapt to a new target category (e.g., cats), the model learns only a small projection matrix. This matrix maps the frozen features of the core network to the specific geometry of the new category, representing only about 10% of the total parameters.
  4. Unsupervised Training: The model is trained using a conditional image generation task. The system tries to reconstruct an image of an object using only the discovered landmarks and a deformed version of the original image. If the landmarks are semantically meaningful and stable, the generator can successfully "un-warp" the image to its original state.

Datasets and Key Insights

The researchers evaluated their method across several diverse datasets to prove its versatility, including Faces (CelebA, MAFL, AFLW, LS3D), Shoes (UT-Zappos50k), and Animals (Cats Head). The core network was primarily pre-trained on the MPII Human Pose dataset.

Several key insights emerged from this approach:

  • Constraints are a Feature, Not a Bug: By freezing the core network and only learning a projection matrix, the optimization problem becomes much more constrained. In unsupervised learning, having "too much freedom" often leads the model to find useless shortcuts. Restricting the learnable parameters forces the model to find stable, human-like structures.
  • Landmarks are Universal: The "semantic essence" of a landmark—being a stable, identifiable point on a shape—is transferable across wildly different categories (e.g., from human joints to cat ears).
  • The "Backward" Test: The authors emphasize that "Forward" error (how well discovered points map to human labels) isn't enough. They highlight "Backward" error (how well human labels map to discovered points) as a better measure of whether the model has found a complete and stable set of landmarks.

Performance and Results

The proposed method outperformed previous unsupervised-from-scratch techniques on the MAFL and AFLW datasets, establishing new state-of-the-art results.

Qualitative comparison showing the proposed method's superior landmark stability on faces and cats under rotation, compared to training from scratch or fine-tuning.
  • Superior Stability: Qualitatively and quantitatively, the landmarks discovered by this method are much more stable under rotation and scaling compared to models trained from scratch or even those that were fully "fine-tuned."
  • Data Efficiency: Because the model leverages prior knowledge, it can learn effectively with much smaller datasets (e.g., 1,000 images) where other methods would typically fail or collapse to a single point.

Limitations and Future Work

While the method performs exceptionally well on 2D images, results on the LS3D dataset show that extreme 3D rotations remain a significant challenge for unsupervised discovery, as landmarks can still "slip" when the viewpoint changes drastically. Additionally, the approach assumes a supervised model is available for at least one source category. The impact of choosing different source categories for various target categories remains an open question. Future work may explore combining this adaptation approach with few-shot learning techniques to further reduce the data requirements for new object categories.

Summary updated

Abstract

This paper proposes a method to ease the unsupervised learning of object landmark detectors. Similarly to previous methods, our approach is fully unsupervised in a sense that it does not require or make any use of annotated landmarks for the target object category. Contrary to previous works, we do however assume that a landmark detector, which has already learned a structured representation for a given object category in a fully supervised manner, is available. Under this setting, our main idea boils down to adapting the given pre-trained network to the target object categories in a fully unsupervised manner. To this end, our method uses the pre-trained network as a core which remains frozen and does not get updated during training, and learns, in an unsupervised manner, only a projection matrix to perform the adaptation to the target categories. By building upon an existing structured representation learned in a supervised manner, the optimization problem solved by our method is much more constrained with significantly less parameters to learn which seems to be important for the case of unsupervised learning. We show that our method surpasses fully unsupervised techniques trained from scratch as well as a strong baseline based on fine-tuning, and produces state-of-the-art results on several datasets. Code can be found at tiny.cc/GitHub-Unsupervised
BibTeX
@inproceedings{sanchez2019object,
  author = {Enrique Sanchez and Georgios Tzimiropoulos},
  title = {{Object landmark discovery through unsupervised adaptation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=WRBzHQe40lI},
}

Loading related content...

Figure