If you train a modern neural network on thousands of unlabeled images of cars, it easily learns a global representation to distinguish a car from a tree. However, if you ask that same network to locate the front-left wheel of every car, it struggles.
Without human labels, establishing semantic correspondence—ensuring that a specific discovered landmark or part index consistently maps to the same anatomical or functional structure across different instances—is deeply challenging. As noted by Yuting Zhang et al. in Unsupervised Discovery of Object Landmarks as Structural Representations, standard deep networks rely on the implicit biases of convolutional layers rather than formal constraints. Consequently, a model left to its own devices might group pixels by local textures or lighting conditions rather than their true semantic identity.
To solve this, researchers have developed frameworks that force networks to discover meaningful parts by introducing structural, geometric, and generative bottlenecks. Here is why they work.
Consistency Through Template Deformation
One of the most intuitive ways to enforce semantic consistency is to frame part discovery as a geometric puzzle. Instead of letting a network select arbitrary points on an object, models can be constrained to deform a shared, canonical "template" to match individual target shapes.
Consider the approach taken by Tomas Jakab et al. in KeypointDeformer: Unsupervised 3D Keypoint Discovery for Shape Control. The authors designed a system where a network must predict a sparse set of keypoints to serve as control handles for warping a source shape into a target shape. This source-to-target alignment task creates a powerful mathematical constraint: if a predicted keypoint randomly jumps from a chair's leg to its backrest across different instances, the deformation mechanism will tear or distort the geometry, incurring a massive reconstruction penalty.
Similarly, in the continuous domain, Jiaolong Yang et al. demonstrated in Deformed Implicit Field: Modeling 3D Shapes With Learned Dense Correspondence that surface normals can act as an unsupervised proxy for semantic alignment. By enforcing a loss that requires corresponding points across different instances to share similar geometric orientations relative to a category template, the model naturally aligns functional parts without requiring a single manual label.
Disentangling Shape and Appearance
When working with 2D images, explicit 3D templates are not always available. However, objects within a category still share a fundamental property: their visual appearance (color, texture, lighting) varies independently of their underlying structural layout.
In Unsupervised Part Segmentation Through Disentangling Appearance and Shape, Shilong Liu et al. leveraged this insight by routing image features through a "squeeze-and-expand" bottleneck designed to separate style from geometry. To prevent the network from taking shortcuts—such as grouping parts purely by color—the authors applied a classification loss directly to the isolated appearance features. This forces the model to map the "wings" or "eyes" of entirely different individuals to a shared, abstract feature space. Because the network cannot use local texture variations as a crutch, it is forced to rely on the shared spatial layout of the object category to achieve clean segmentations.
Hijacking Emergent Semantics in Foundation Models
Recently, the strategy has shifted from training specialized constraints from scratch to extracting the rich, emergent semantic properties already embedded within foundation models. Large-scale generative models (like Stable Diffusion) and self-supervised vision models (like DINO) must inherently understand object parts to successfully generate or contrast images.
For example, Eric Hedlin et al. showed in Unsupervised Keypoints from Pretrained Diffusion Models that optimizing text embeddings to produce localized attention maps can surface highly stable landmarks across diverse, "in-the-wild" datasets. Because the underlying foundation model has already seen millions of images, its internal attention maps naturally maintain semantic identity across extreme variations in pose and lighting.
Taking this a step further, Yuval Alaluf et al. demonstrated in Cross-Image Attention for Zero-Shot Appearance Transfer that swapping the internal keys and values between different images during a diffusion model's denoising process enables high-fidelity appearance transfer. This zero-shot phenomenon works precisely because the network’s internal cross-image attention layers implicitly map the structure of one object instance directly onto another, providing a ready-made mechanism for semantic alignment without any additional training.
Key Foundational Work
Unsupervised Learning of Human Motion Models (2001) — contributed 1 concept elements to this direction.