ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors

Xiaoxuan Ma, Stephan P. Kaufhold, Jiajun Su, Wentao Zhu, Jack Terwilliger, Andres Meza, Yixin Zhu, Federico Rossano, Yizhou Wang

Abstract

ChimpACT is a large-scale longitudinal dataset designed to advance computer vision research in non-human primate behavior. Comprising over 160,000 frames recorded over four years, it features expert-verified annotations for multi-object tracking, pose estimation, and fine-grained social action recognition. By capturing the developmental trajectory and social dynamics of a chimpanzee group in a semi-natural environment, ChimpACT addresses the limitations of existing datasets and provides a challenging benchmark for models to transition from human-centric tasks to complex animal social modeling.

Introduction

While computer vision has made massive strides in understanding human behavior, our closest living relatives—non-human primates—have been largely overlooked. Existing primate datasets are often limited by a lack of temporal context, focus on individuals in constrained indoor environments, or lack the fine-grained labels necessary to understand complex social interactions.

This gap hinders research in evolutionary biology, animal welfare, and social modeling. To truly understand the evolution of sociality, researchers require the ability to track individuals over long periods (longitudinally) in semi-natural environments. ChimpACT provides a high-quality, multi-task dataset that captures the developmental trajectory and social tapestry of a chimpanzee group.

Methodology

The researchers collected 405 hours of video at the Leipzig Zoo, Germany, between 2015 and 2018. They employed focal sampling, specifically following a juvenile male named Azibo from birth through his early years.

To process this footage into a machine-learning-ready format, the team:

  1. Curated 163 clips (approximately 1,000 frames each).
  2. Annotated Bounding Boxes and IDs for over 23 unique individuals.
  3. Defined 16 Keypoints for pose estimation, including facial markers like the upper and lower lips to capture eating or vocalizing behaviors.
  4. Developed a Hierarchical Ethogram: A structured dictionary of behaviors categorized into Locomotion, Object Interaction, Social Interaction, and Others, further subdivided into 23 nuanced actions (e.g., "grooming" vs. "being groomed").
The ChimpACT ethogram organizes behaviors into a hierarchy, while the kinship chart tracks the social group's relationships.

The ChimpACT Dataset

ChimpACT distinguishes itself from existing benchmarks like AP-10K or AnimalKingdom by providing a "triple threat" of annotations: tracking, pose, and spatiotemporal action labels, all within a longitudinal context.

The dataset features:

  • 160,500 frames across 163 videos.
  • 56,324 pose annotations.
  • 64,289 action labels.
  • A mix of indoor and outdoor environments with varying lighting and complex backgrounds.
Example of multi-task annotations in ChimpACT, including bounding boxes, individual IDs, pose skeletons, and action labels.

Challenges and Insights

  • The "Human Gap": Models trained on human data do not seamlessly transfer to chimpanzees. Chimpanzees possess significantly higher joint flexibility and a wider range of motion. Additionally, their dark, textured fur presents a "low-contrast" challenge for detection and pose estimation compared to human skin.
  • Long-Tail Distribution: Reflecting real-world behavior, the dataset is highly imbalanced. Common actions like "resting" or "moving" dominate, while critical social behaviors like "embracing" or "begging" are rare, making ChimpACT an excellent testbed for few-shot learning.
  • Social Reciprocity: By annotating both the "actor" and the "receiver" (e.g., grooming vs. being groomed), the dataset enables models to learn the nuances of social exchange rather than just isolated movements.

Benchmarking Results

The authors benchmarked several state-of-the-art (SOTA) models across three tracks:

  1. Tracking & ReID: QDTrack was the top performer (50.3 HOTA), though it struggled with identity switches during heavy occlusion.
  2. Pose Estimation: Heatmap-based models like DarkPose with an HRNet backbone performed best. Accuracy was notably higher for facial keypoints (eyes/lips) than for limbs, likely due to the distinctive visual patterns of the face compared to fur-covered extremities.
  3. Action Detection: Models like SlowFast achieved decent results for simple locomotion (mAP ~60) but struggled significantly with complex social interactions. The "others" category, which includes rare behaviors, registered a near-zero mAP, highlighting the difficulty of the task.

Conclusion

While ChimpACT provides a robust benchmark, it is based on a captive population in a semi-natural environment, lacking certain wild behaviors like complex foraging or intergroup conflict. Furthermore, the focus on a single juvenile individual introduces a bias toward his specific interactions. Future work will involve expanding the dataset to include more individuals and developing algorithms capable of handling the extreme occlusions and "self-occlusions" (e.g., huddling) common in primate groups.

Summary updated

BibTeX
@inproceedings{ma2023chimpact,
  author = {Xiaoxuan Ma and Stephan P. Kaufhold and Jiajun Su and Wentao Zhu and Jack Terwilliger and Andres Meza and Yixin Zhu and Federico Rossano and Yizhou Wang},
  title = {{ChimpACT: A Longitudinal Dataset for Understanding Chimpanzee Behaviors}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=44vmw1sOtr},
}
Figure