Abstract
Current image-based keypoint detection methods for animal bodies and faces are generally divided into fully-supervised and few-shot class-agnostic approaches. The former relies on laborious manual annotations, while the latter requires annotated support images at test time. To realize zero-shot keypoint detection without prior annotation, we introduce the Open-Vocabulary Keypoint Detection (OVKD) task, designed to use text prompts for identifying arbitrary keypoints across any species. We propose a novel framework, Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM), which synergistically combines vision and language models. KDSM integrates Domain Distribution Matrix Matching (DDMM) and a Vision-Keypoint Relational Awareness (VKRA) module to improve generalizability. Our experiments demonstrate that KDSM significantly outperforms baselines and achieves results comparable to state-of-the-art few-shot methods in a zero-shot fashion.
Introduction
Traditional animal keypoint detection faces a significant scaling challenge. Fully supervised methods require thousands of manual annotations for every new species, a task that becomes increasingly impractical given the vast diversity of the animal kingdom. Few-shot approaches attempt to mitigate this by using a small number of "support images" as references; however, they still necessitate annotated examples during inference.
This research identifies a critical gap: the potential to leverage semantic understanding for detection. If a model understands the concept of an "eye" and the visual features of a "giraffe," it should theoretically be able to locate a giraffe's eye without prior labeled examples. We formalize this as the Open-Vocabulary Keypoint Detection (OVKD) task, moving keypoint detection into the realm of zero-shot, language-driven computer vision.
Methodology: The KDSM Framework
The proposed KDSM (Open-Vocabulary Keypoint Detection with Semantic-feature Matching) framework moves beyond simple CLIP-based baselines by introducing two sophisticated modules designed to handle the complexity of arbitrary species and anatomical parts.
Vision-Keypoint Relational Awareness (VKRA)
This module utilizes Transformer-based self-attention and cross-attention mechanisms. Rather than analyzing pixels in isolation, VKRA forces the model to learn the interplay between language features and local visual regions. This helps the system "understand" the spatial context—where a specific semantic part (like a "nose tip") should reside relative to the overall body structure.
Domain Distribution Matrix Matching (DDMM)
To handle the virtually infinite combinations of species and keypoints, KDSM avoids dedicated output channels for every pair. Instead, DDMM uses K-means clustering on text embeddings to group semantically similar keypoints (e.g., "left eye" across various mammals) into shared heatmap channels. During inference, the model predicts a distribution matrix to map the specific text prompt to the correct heatmap channel, effectively solving the "label explosion" problem.
Experiments and Results
Datasets and Setup
The model was evaluated on several benchmarks:
- MP-100: A standard dataset for category-agnostic pose estimation.
- MP-78: A curated subset of MP-100 focusing on 78 categories with clear semantic names (e.g., "upper lip"), utilizing ChatGPT to generate precise anatomical descriptions.
- AnimalWeb: Used to test "long-tail" performance on rare animal species.
Performance Metrics
KDSM demonstrated significant improvements over existing baselines and competitive performance against few-shot models:
- Zero-Shot Superiority: In settings detecting new keypoint types, KDSM achieved a Mean [email protected] of 88.23, vastly outperforming a simple CLIP baseline (42.93).
- Efficiency: Operating at approximately 65 FPS on an A100 GPU, KDSM is significantly faster than top-tier few-shot methods (15–50 FPS) because it eliminates the overhead of processing support images.
- Generalization: On the AnimalWeb dataset, KDSM maintained high accuracy even for rare species, proving that language serves as an effective bridge for knowledge transfer across species.
Discussion and Limitations
While KDSM represents a leap forward in zero-shot detection, challenges remain. Environmental factors such as extreme occlusion, poor lighting, and low resolution can still degrade performance. Additionally, predicted heatmaps occasionally suffer from "anisotropic Gaussian distribution" issues, where the predicted points appear stretched or skewed.
Future work will explore more robust loss functions, such as STAR loss, to address these spatial distortions. Furthermore, while the current framework relies on semantic text prompts, future iterations may explore hybrid prompts that combine text with minimal visual hints (like a single click) to resolve ambiguity in complex scenes.
Summary updated