Abstract
Quantification of behavior is critical in applications ranging from neuroscience, veterinary medicine and animal conservation efforts. A common key step for behavioral analysis is first extracting relevant keypoints on animals, known as pose estimation. However, reliable inference of poses currently requires domain knowledge and manual labeling effort to build supervised models. We present a series of technical innovations that enable a new method, collectively called SuperAnimal, to develop and deploy deep learning models that require zero additional human labels and model training. SuperAnimal allows video inference on over 45 species with only two global classes of animal pose models. If the models need fine-tuning, we show SuperAnimal models are 10$\times$ more data efficient and outperform prior transfer-learning-based approaches. Moreover, we provide an unsupervised video-adaptation method to refine keypoints in videos. We illustrate the utility of our model in behavioral classification in mice and gait analysis in horses. Collectively, this presents a data-efficient solution for animal pose estimation for downstream behavioral analysis.
Introduction
In fields such as neuroscience and conservation biology, tracking animal movement—pose estimation—is a fundamental prerequisite for behavioral analysis. Traditionally, tools like DeepLabCut have revolutionized this space but still impose a significant "manual labeling bottleneck." Researchers often must label hundreds of images for every new species, camera angle, or experimental setup. This process is not only time-consuming but also suffers from inconsistent "vocabularies" across labs (e.g., naming a point "snout" vs. "nose") and disjoint datasets where only subsets of body parts are labeled.
SuperAnimal addresses these challenges by introducing "foundation models" for animal pose estimation. These models are designed to be "panoptic," unifying diverse datasets into a single keypoint space. The result is a system capable of zero-shot performance across more than 45 species, drastically reducing or eliminating the need for manual annotation.
Methods: Building a Unified Pose Estimator
The SuperAnimal framework relies on several technical innovations to overcome the limitations of traditional supervised learning.
Generalized Data Conversion and Shared Vocabulary
To handle the "vocabulary mess," the authors developed a Generalized Data Converter. This tool maps inconsistent labels from disparate sources into a unified, panoptic vocabulary. This allows the model to learn from a vast array of existing datasets without being confused by differing nomenclature.
Keypoint Gradient Masking
Training on disjoint datasets (where one set labels tails and another only faces) typically causes issues. SuperAnimal uses Keypoint Gradient Masking, a technique that allows the model to learn from incomplete labels. If a specific keypoint is missing in a training image, the model masks the loss for that point, preventing the network from being penalized for "missing" information it was never given.
Memory Replay and Video Adaptation
To adapt models to specific datasets without losing general knowledge (catastrophic forgetting), the authors employ Memory Replay Fine-tuning. This uses "pseudo-labels" from the base model to anchor the training process. Furthermore, an unsupervised video-adaptation method utilizes spatial-pyramid search and pseudo-labeling to automatically refine keypoint predictions in new videos without human intervention.
Results: Zero-Shot Performance and Data Efficiency
The SuperAnimal models were trained on massive new datasets: TopViewMouse-5K (merging 13 overhead mouse datasets) and Quadruped-80K (over 85,000 images of 45+ species).
Zero-Shot Capabilities
The most significant result is the models' ability to perform "zero-shot" inference. On entirely new datasets, SuperAnimal often matches or exceeds the performance of models specifically trained on those datasets with human labels. This is achieved by leveraging "pose priors"—knowledge of animal geometry learned during pre-training—which proves far more effective than standard ImageNet-based priors.
10-100x Data Efficiency
When fine-tuning is necessary for highly specific environments, SuperAnimal is remarkably efficient. Benchmarks on DLC-Openfield and TriMouse show that SuperAnimal models reach high accuracy with 10–100× less data than standard approaches. For instance, a model fine-tuned on just 10 frames can match the accuracy of a traditional model trained on 100 frames.
Downstream Applications
The utility of these zero-shot keypoints was validated through complex behavioral tasks. The researchers successfully used SuperAnimal outputs to classify "rearing" behavior in mice and perform detailed gait analysis in horses, identifying stride and swing phases without any task-specific training for the pose estimator.
Conclusion and Future Directions
SuperAnimal represents a shift toward foundation models in animal behavior research, effectively ending the manual labeling bottleneck for many common species. While currently focused on mammals, future expansions of the "Model Zoo" will include birds, insects, and fish. The integration of unsupervised keypoint discovery remains a promising avenue to further enhance the model's ability to adapt to novel morphologies without human ground truth.
Source: Ye, S., et al. "SuperAnimal pretrained pose estimation models for behavioral analysis." arXiv:2203.07436v4, 2023.
Summary updated