Abstract
Category-Agnostic Pose Estimation (CAPE) is a challenging task that requires localizing keypoints on novel object categories using only a few annotated support images. Current approaches often struggle with overfitting to specific support image textures and resolving ambiguities between similar-looking parts. We introduce ESCAPE, a Bayesian framework that leverages "super-keypoints"—abstract, high-level representations of semantically related features shared across different species or objects. By treating these super-keypoints as a learned prior, ESCAPE provides a robust "starting guess" for novel categories. Our method employs a Learnable Matching Network and an Expectation-Maximization (EM) adaptation strategy to achieve state-of-the-art performance on the MP-100 benchmark, significantly outperforming existing methods in few-shot scenarios.
Introduction
Traditional 2D pose estimation models are typically category-specific specialists. While they excel at identifying joints on humans or parts on vehicles, they fail when presented with novel categories, such as rare animal species. Category-Agnostic Pose Estimation (CAPE) seeks to bridge this gap by creating "generalist" models capable of adapting to any category given minimal support examples.
The primary challenges in CAPE are twofold:
- Overfitting: With limited data (1-5 examples), models tend to memorize specific textures of support images rather than learning the underlying geometry.
- Keypoint Ambiguity: Distinguishing between similar-looking parts (e.g., left vs. right limbs) is difficult without explicit labels or structural context.
Methodology: Encoding Super-keypoints
ESCAPE operates on the principle that many keypoints across disparate categories share semantic similarities.
Discovering Super-keypoints
During training on "base" categories (e.g., antelope, deer, zebra), the model employs a clustering algorithm to identify groups of similar keypoints. These clusters form super-keypoints—mathematical prototypes representing abstract concepts like "nose" or "hind leg" that transcend specific species.
Bayesian Prior and EM Adaptation
ESCAPE treats these super-keypoints as a Bayesian prior. When encountering a novel category, the model uses these priors as a "starting guess" to ground predictions in realistic part distributions. The adaptation process follows an Expectation-Maximization (EM) approach:
- E-step: Matches novel keypoints to the most relevant super-keypoints using a Learnable Matching Network. This network utilizes self- and cross-attention to consider the global structure of the object, making matches more reliable than simple feature comparisons.
- M-step: Calculates final keypoint prototypes by blending features from support images with the knowledge encoded in the matched super-keypoints via Maximum A Posteriori (MAP) estimation.
Key Insights
A critical observation in this work is that semantic similarity is universal across categories. Visualization of the feature space reveals that a model trained on base categories like zebras already possesses a coherent representation of where a novel category's (e.g., a horse) keypoints should reside.
By "pulling" estimated keypoint prototypes toward these learned super-keypoints, the model becomes robust to the noise and variations inherent in few-shot support images. Furthermore, the use of a transformer-based matching network allows the model to resolve local ambiguities by considering the entire object's geometry.
Experimental Results
The researchers evaluated ESCAPE on the MP-100 (Multi-category Pose) benchmark, which spans over 20,000 instances across 100 categories, including humans, animals, furniture, and vehicles. The dataset is split into 70 base categories for training and 20 novel categories for testing generalizability.
ESCAPE achieved new state-of-the-art (SOTA) performance:
- 1-Shot Performance: 82.74% Mean PCK (Percentage of Correct Keypoints), outperforming the previous best (CapeFormer) by approximately 2.5%.
- 5-Shot Performance: 87.63% Mean PCK.
Notably, ESCAPE does not require pre-defined keypoint identifiers, making it highly practical for open-world applications where users define arbitrary keypoints.
Conclusion and Future Work
While ESCAPE demonstrates significant improvements, its performance is tied to the diversity of the base categories used for initial clustering. If base categories do not sufficiently cover the diversity of novel categories, the super-keypoint priors may be less effective. Future work may involve integrating more powerful foundation model feature extractors (e.g., DINOv2 or DIFT) to further enhance generalizability. Despite these dependencies, ESCAPE provides a robust framework for few-shot pose estimation by effectively leveraging cross-category semantic priors.
Summary updated