In computer vision, a fundamental gap exists between what a neural network sees and what a human understands. While a standard Convolutional Neural Network (CNN) can identify a "bird," its internal filters often respond to a chaotic mix of textures and edges that don't necessarily correspond to a "beak" or a "wing." This lack of semantic consistency—the ability to recognize the same functional part across different individuals, poses, and lighting—is the primary hurdle for tasks like fine-grained recognition, motion transfer, and 3D reconstruction.
Recent research has converged on a shared goal: discovering these meaningful parts and keypoints entirely from unlabeled data. By forcing models to find structures that are not just geometrically stable but semantically invariant across a dataset, researchers are building a bridge between raw pixels and structured object understanding.
The Problem of Semantic Drift
If you train a model to find "interesting points" on a face without supervision, it might consistently pick the tip of the nose. However, if the face turns or the lighting changes, an unsupervised model often suffers from "semantic drift." A filter that responded to an eye in one image might respond to a shadow or a patch of hair in another.
As Quanshi Zhang and colleagues observed in their work on Explanatory Graphs, raw feature maps in pre-trained networks are often "entangled." To solve this, researchers seek to disentangle these filters into a structured dictionary of parts. The intuition is that a "part" is not just a visual pattern, but a functional component that should behave predictably across the entire category.
Discovery through Factorization and Co-occurrence
One of the most intuitive ways to find parts is to look for things that frequently appear together. Edo Collins and Radhakrishna Achanta demonstrated this with Deep Feature Factorization. By applying Non-negative Matrix Factorization (NMF) to CNN activations, they treated deep features as a "bag of concepts."
Because NMF decomposes a matrix into additive parts, the model naturally clusters activations that co-occur. If "beak-like" features and "eye-like" features always appear in bird images but in different spatial locations, the factorization separates them into distinct, semantically consistent components. This allows the model to localize similar structures across different images without ever being told what a "beak" is.
Equivariance: Parts Should Move Predictably
A key mechanism for ensuring these parts are semantically "real" is equivariance. If an object rotates or scales, its parts should move in a mathematically predictable way.
In Unsupervised learning of object landmarks by factorized spatial embeddings, the authors forced a network to detect points that move consistently with known image transformations. If the model picks a point on a shoe and the image is flipped, the model is penalized unless it picks the same point on the flipped shoe. This constraint forces the network to ignore transient features (like shadows) and latch onto stable, anatomical landmarks.
Similarly, Ruoxi Shi and Yang You extended this to 3D with Skeleton Merger. By reconstructing 3D objects from a "skeleton" of edges rather than just sparse points, they ensured that discovered keypoints (like an airplane's wingtip) remained semantically aligned across different models in a category.
Talking to Foundation Models
The latest frontier in part discovery involves "interrogating" large, pre-trained foundation models like Stable Diffusion. These models already possess a deep, latent understanding of the world from being trained on billions of images.
Eric Hedlin and his team found that they could extract Unsupervised Keypoints from Pretrained Diffusion Models by optimizing "dummy" text embeddings. By asking the model to produce a localized attention map for a specific (but unnamed) concept, they can find points that are semantically consistent even in messy, "in-the-wild" data. Satyajit Tourani furthered this by using a "rotate-and-render" strategy, leveraging the model's internal 3D-aware representations to discover facial landmarks without a single human label.
Why It Works: The Canonical Template
The underlying reason these methods succeed is the implicit learning of a canonical template. Whether it is Jiaolong Yang's Deformed Implicit Field or Bin Li's Global Object-Centric Learning, the models are essentially learning a "mental library" of what a category looks like in a neutral, un-deformed state.
By mapping every individual instance (a specific chair or a specific bird) back to this shared coordinate system, the models can establish dense semantic correspondence. Once the model knows how a specific chair relates to the "canonical chair," it can transfer textures, labels, or motions between any two instances in the dataset. This turns unsupervised learning from a simple clustering task into a sophisticated understanding of object geometry and function.