Language-driven Open-Vocabulary Keypoint Detection for Animal Body and Face

Hao Zhang, Kaipeng Zhang, Lumin Xu, Shenqi Lai, Wenqi Shao, Nanning Zheng, Ping Luo, Yu Qiao

Abstract

Current image-based keypoint detection methods for animal bodies and faces are generally divided into fully-supervised and few-shot class-agnostic approaches. The former relies on laborious manual annotations, while the latter requires annotated support images at test time. To realize zero-shot keypoint detection without prior annotation, we introduce the Open-Vocabulary Keypoint Detection (OVKD) task, designed to use text prompts for identifying arbitrary keypoints across any species. We propose a novel framework, Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM), which synergistically combines vision and language models. KDSM integrates Domain Distribution Matrix Matching (DDMM) and a Vision-Keypoint Relational Awareness (VKRA) module to improve generalizability. Our experiments demonstrate that KDSM significantly outperforms baselines and achieves results comparable to state-of-the-art few-shot methods in a zero-shot fashion.

Introduction

Traditional animal keypoint detection faces a significant scaling challenge. Fully supervised methods require thousands of manual annotations for every new species, a task that becomes increasingly impractical given the vast diversity of the animal kingdom. Few-shot approaches attempt to mitigate this by using a small number of "support images" as references; however, they still necessitate annotated examples during inference.

Comparison between few-shot class-agnostic keypoint detection (a) and the proposed language-driven open-vocabulary keypoint detection task (b).

This research identifies a critical gap: the potential to leverage semantic understanding for detection. If a model understands the concept of an "eye" and the visual features of a "giraffe," it should theoretically be able to locate a giraffe's eye without prior labeled examples. We formalize this as the Open-Vocabulary Keypoint Detection (OVKD) task, moving keypoint detection into the realm of zero-shot, language-driven computer vision.

Methodology: The KDSM Framework

The proposed KDSM (Open-Vocabulary Keypoint Detection with Semantic-feature Matching) framework moves beyond simple CLIP-based baselines by introducing two sophisticated modules designed to handle the complexity of arbitrary species and anatomical parts.

The detailed architecture of the KDSM framework, featuring the Relational Awareness Module and Domain Distribution Matrix Matching (DDMM).

Vision-Keypoint Relational Awareness (VKRA)

This module utilizes Transformer-based self-attention and cross-attention mechanisms. Rather than analyzing pixels in isolation, VKRA forces the model to learn the interplay between language features and local visual regions. This helps the system "understand" the spatial context—where a specific semantic part (like a "nose tip") should reside relative to the overall body structure.

Domain Distribution Matrix Matching (DDMM)

To handle the virtually infinite combinations of species and keypoints, KDSM avoids dedicated output channels for every pair. Instead, DDMM uses K-means clustering on text embeddings to group semantically similar keypoints (e.g., "left eye" across various mammals) into shared heatmap channels. During inference, the model predicts a distribution matrix to map the specific text prompt to the correct heatmap channel, effectively solving the "label explosion" problem.

Experiments and Results

Datasets and Setup

The model was evaluated on several benchmarks:

  • MP-100: A standard dataset for category-agnostic pose estimation.
  • MP-78: A curated subset of MP-100 focusing on 78 categories with clear semantic names (e.g., "upper lip"), utilizing ChatGPT to generate precise anatomical descriptions.
  • AnimalWeb: Used to test "long-tail" performance on rare animal species.

Performance Metrics

KDSM demonstrated significant improvements over existing baselines and competitive performance against few-shot models:

  • Zero-Shot Superiority: In settings detecting new keypoint types, KDSM achieved a Mean [email protected] of 88.23, vastly outperforming a simple CLIP baseline (42.93).
  • Efficiency: Operating at approximately 65 FPS on an A100 GPU, KDSM is significantly faster than top-tier few-shot methods (15–50 FPS) because it eliminates the overhead of processing support images.
  • Generalization: On the AnimalWeb dataset, KDSM maintained high accuracy even for rare species, proving that language serves as an effective bridge for knowledge transfer across species.

Discussion and Limitations

While KDSM represents a leap forward in zero-shot detection, challenges remain. Environmental factors such as extreme occlusion, poor lighting, and low resolution can still degrade performance. Additionally, predicted heatmaps occasionally suffer from "anisotropic Gaussian distribution" issues, where the predicted points appear stretched or skewed.

Future work will explore more robust loss functions, such as STAR loss, to address these spatial distortions. Furthermore, while the current framework relies on semantic text prompts, future iterations may explore hybrid prompts that combine text with minimal visual hints (like a single click) to resolve ambiguity in complex scenes.

Summary updated

Abstract

Current image-based keypoint detection methods for animal (including human) bodies and faces are generally divided into full-supervised and few-shot class-agnostic approaches. The former typically relies on laborious and time-consuming manual annotations, posing considerable challenges in expanding keypoint detection to a broader range of keypoint categories and animal species. The latter, though less dependent on extensive manual input, still requires necessary support images with annotation for reference during testing. To realize zero-shot keypoint detection without any prior annotation, we introduce the Open-Vocabulary Keypoint Detection (OVKD) task, which is innovatively designed to use text prompts for identifying arbitrary keypoints across any species. In pursuit of this goal, we have developed a novel framework named Open-Vocabulary Keypoint Detection with Semantic-feature Matching (KDSM). This framework synergistically combines vision and language models, creating an interplay between language features and local keypoint visual features. KDSM enhances its capabilities by integrating Domain Distribution Matrix Matching (DDMM) and other special modules, such as the Vision-Keypoint Relational Awareness (VKRA) module, improving the framework's generalizability and overall performance.Our comprehensive experiments demonstrate that KDSM significantly outperforms the baseline in terms of performance and achieves remarkable success in the OVKD task.Impressively, our method, operating in a zero-shot fashion, still yields results comparable to state-of-the-art few-shot species class-agnostic keypoint detection methods.We will make the source code publicly accessible.
BibTeX
@article{zhang2023language,
  author = {Hao Zhang and Kaipeng Zhang and Lumin Xu and Shenqi Lai and Wenqi Shao and Nanning Zheng and Ping Luo and Yu Qiao},
  title = {{Language-driven Open-Vocabulary Keypoint Detection for Animal Body and Face}},
  journal = {Int. J. Comput. Vis.},
  year = {2023},
  url = {https://openreview.net/forum?id=aS35UYCcMI},
}
Figure