Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals

Yaning Han, Ke Chen, Yunke Wang, Wenhao Liu, Xiaojing Wang, Jiahui Liao, Yi-Ting Huang, Chuanliang Han, Kang Huang, Jiajia Zhang, Shengyuan Cai, Zhouwei Wang, Yongji Wu, Gao Gao, Nan Wang, Jinxiu Li, Yangwangzi Song, Jing Li, Guodong Wang, Liping Wang, Ya‐Ping Zhang, Pengfei WeibioRxiv 2023 · DOI

TL;DR

This paper introduces the Social Behavior Atlas (SBeA), a novel computational framework that automates the 3D tracking and mapping of social interactions among free-moving animals. SBeA overcomes limitations of existing methods by reducing manual annotation needs, handling occlusions with multi-camera setups, and accurately identifying individual animals, enabling more detailed and reliable studies of animal social behavior.

What Problem Does This Paper Address?

Studying animal social behavior is crucial for understanding survival, reproduction, and even neurological disorders. However, accurately tracking and analyzing the complex, close-range interactions of multiple free-moving animals has been a significant challenge. Existing methods often require extensive manual annotation, struggle with occlusions (when animals block each other's view), and have difficulty distinguishing between similar-looking individuals over time. This limits the depth and scale of research in fields like neuroscience and ecology.

Methods

SBeA tackles these challenges with a multi-stage approach:

  • Deep Learning-based Video Instance Segmentation (VIS): SBeA uses a novel approach involving "continuously occluded copy-paste" data augmentation to generate synthetic scenarios with occlusions. This allows for training a VIS model (specifically, a modified VisTR) with less manual annotation, improving its ability to segment individual animals even when they are partially hidden.
  • 3D Pose Reconstruction: Leveraging a multi-camera setup (MouseVenue3D system), SBeA reconstructs the 3D poses of animals by combining 2D pose information from multiple views and using geometric constraints to minimize reprojection errors.
  • Unsupervised Dynamic Behavioral Clustering: SBeA decomposes animal trajectories into locomotion, non-locomotor movement, and body distance components. These are then merged into social behavior modules. These modules are embedded into a low-dimensional space using UMAP and ResMLP for feature encoding, creating a "social behavior atlas." This atlas allows for the identification of distinct behavioral patterns and subtle abnormalities.
  • Bidirectional Transfer Learning for Identification: To address the challenge of identifying individual animals, SBeA employs bidirectional transfer learning. Segmentation models trained for multi-animals can be transferred to single-animal segmentation, and vice-versa for identification models. This significantly reduces the need for manual data annotation for identity recognition.

Datasets

  • SBM-VIS Dataset: A dataset of free-social behavior of C57BL/6 mice, captured using the MouseVenue3D system. It includes annotated frames for training VIS models.
  • PAIR-R24M Dataset: A dataset containing 3D poses, social behavior labels, and subject behavior labels of rats, used for supervised validation of SBeA.
  • Custom Datasets: The paper also describes the creation of custom datasets for training and evaluating SBeA on mice, birds, and dogs, with varying experimental setups (e.g., different open field sizes).

Key Insights and Ideas

  • Data Augmentation for Occlusion: The "continuously occluded copy-paste" technique is a clever way to generate realistic training data for handling occlusions without requiring extensive manual labeling of occluded scenarios.
  • Reduced Annotation Burden: By combining advanced deep learning techniques with smart data augmentation and transfer learning, SBeA drastically cuts down the manual annotation effort compared to previous methods.
  • Unified Framework: SBeA integrates pose tracking, identification, and behavior mapping into a single, cohesive framework, providing a more holistic view of social interactions.
  • Subtle Behavior Detection: The ability to identify subtle behavioral abnormalities, as demonstrated with Shank3B knockout mice, highlights SBeA's power in detecting nuanced social deficits.
  • Cross-Species Generalizability: The successful application to mice, birds, and dogs demonstrates the robustness and versatility of the SBeA framework.

Key Results

  • Reduced Annotation: SBeA reduces manual annotation labor by up to 50% compared to traditional pose estimation methods.
  • High Identification Precision: SBeA achieves over 90% identification precision without human data annotation for multi-animal identification.
  • Accurate Tracking: The system demonstrates high tracking likelihood for both birds and dogs, achieving satisfactory tracking precision.
  • Shank3B KO Mice: SBeA successfully identified subtle social interaction abnormalities in Shank3B knockout mice, differentiating them from wild-type mice.
  • Cross-Species Performance: SBeA identified 34 social behavior classes in birds and 15 in dogs, showcasing its applicability across species.
  • RMSE Comparison: SBeA showed significantly lower RMSE (prediction error) compared to maDLC for most body parts, especially in close contact scenarios.

Limitations and Future Work

  • Storage Consumption: The authors suggest future work on developing an end-to-end model to reduce storage consumption, potentially by integrating data generation into the VIS model.
  • Algorithmic Bridge: They propose creating an algorithmic bridge between single-animal and multi-animal behavior atlases to facilitate broader analysis.
  • Real-time Performance: While not explicitly stated as a limitation, the computational demands of such a system suggest that further optimization for real-time applications might be an area for future research.

Summary updated

Abstract

Abstract The study of social behaviors in animals is essential for understanding their survival and reproductive strategies. However, accurately tracking and analyzing the social interactions of free-moving animals has remained a challenge. Existing multi-animal pose estimation techniques suffer from drawbacks such as the need for extensive manual annotation and difficulty in discriminating between similar-looking animals in close social interactions. In this paper, we present the Social Behavior Atlas (SBeA), a novel computational framework that solves these challenges by employing a deep learning-based video instance segmentation model, 3D pose reconstruction, and unsupervised dynamic behavioral clustering. SBeA framework also involves a multi-camera setup to prevent occlusion, and a novel approach to identify individual animals in close social interactions. We demonstrate the effectiveness of SBeA in tracking and mapping the 3D close interactions of free-moving animals using the example of genetic mutant mice, birds, and dogs. Our results show that SBeA is capable of identifying subtle social interaction abnormalities, and the models and frameworks developed can be applied to a wide range of animal species. SBeA is a powerful tool for researchers in the fields of neuroscience and ecology to study animal social behaviors with a high degree of accuracy and reliability.
BibTeX
@misc{han2023social,
  author = {Yaning Han and Ke Chen and Yunke Wang and Wenhao Liu and Xiaojing Wang and Jiahui Liao and Yi-Ting Huang and Chuanliang Han and Kang Huang and Jiajia Zhang and Shengyuan Cai and Zhouwei Wang and Yongji Wu and Gao Gao and Nan Wang and Jinxiu Li and Yangwangzi Song and Jing Li and Guodong Wang and Liping Wang and Ya‐Ping Zhang and Pengfei Wei},
  title = {{Social Behavior Atlas: A computational framework for tracking and mapping 3D close interactions of free-moving animals}},
  year = {2023},
  doi = {10.1101/2023.03.05.531235},
  howpublished = {bioRxiv},
}
Figure