Leaving Flatland: Advances in 3D behavioral measurement

Jesse D. Marshall, Tianqing Li, Joshua H. Wu, Timothy Dunn

Abstract

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.

Introduction: The Limitations of "Flatland"

For decades, behavioral science has been largely confined to the two-dimensional plane of standard video recordings. While 2D analysis has provided foundational insights, it suffers from two fundamental flaws: pose ambiguity and pose divergence. Pose ambiguity occurs when multiple distinct 3D configurations appear identical from a single 2D perspective. Conversely, pose divergence describes how the same 3D behavior can look entirely different depending on the camera's viewing angle.

These limitations hinder the reproducibility of behavioral data across laboratories and prevent the capture of true kinematics. To bridge the gap between neural activity and physical movement, researchers must move beyond pixels and measure behavior in real-world units like millimeters. This transition is essential for screening subtle motor deficits in disease models and understanding the precise mechanics of movement.

Methodologies for 3D Measurement

The transition to 3D tracking is driven by two primary technological streams: specialized hardware and deep-learning-based computer vision.

Hardware-Based Direct Measurement

  • Motion Capture (Mocap): Long considered the "gold standard," Mocap uses retroreflective markers and infrared cameras to triangulate positions with micron-level precision. However, its use is often restricted by the need for physical markers, which can be cumbersome or easily removed by smaller animals.
  • Depth Cameras: Utilizing Time-of-Flight or structured light, these sensors generate "depth maps." While effective for top-down views, they often struggle with self-occlusion—for instance, when a rodent’s limbs are hidden beneath its body.

Software-Based Computer Vision

The "deep learning revolution" has introduced several robust alternatives to hardware-heavy setups:

  • Multi-View Triangulation: Tools like DeepLabCut-3D detect 2D keypoints from multiple synchronized cameras and mathematically triangulate them into 3D space.
  • Volumetric CNNs: Models such as DANNCE unproject 2D features into a 3D volume. By learning spatial relationships directly in 3D, these models are significantly more resilient to occlusions than simple triangulation.
  • Monocular Lifting: Using prior knowledge of skeletal structures, models like Liftpose3D can predict 3D coordinates from a single 2D camera view, effectively "lifting" the data into the third dimension.
  • Parametric Shape Models: Moving beyond point-tracking, models like SMPL (for humans) and SMAL (for animals) capture full-body meshes, allowing for the analysis of continuous shape and volume variations.

The Data Revolution

The efficacy of these deep learning models relies on massive, high-quality datasets. Recent contributions have expanded the scope of available training data significantly:

  • Rat7M: 7 million labeled frames of individual rats.
  • PAIR-R24M: 24 million labeled frames specifically designed for multi-animal interaction tracking.
  • OpenMonkeyPose: Over 33,000 human-annotated frames for tracking macaques in complex environments.
  • AcinoSet: Specialized data for tracking cheetahs in the wild.

Furthermore, the use of Synthetic Data—computer-generated imagery (CGI)—is an emerging trend. This allows researchers to train models on vast distributions of species and environments where manual ground-truth labeling would be impossible to obtain.

Key Insights and Applications

The shift to 3D measurement provides more than just better coordinates; it offers a new lens through which to view biology.

Metric Consistency and Neural Logic

3D tracking ensures that data is reported in physical units (mm), making results consistent across different experimental setups and laboratories. This precision has revealed that neurons in the motor cortex are specifically tuned to joint angles of the head and trunk—nuances that were previously obscured in 2D studies. Researchers are also utilizing "weak supervision," where projective geometry serves as a loss function to correct 3D pose predictions without requiring additional manual labels.

Superior Phenotyping

In pharmacological research, 3D-based classifiers like MoSeq have demonstrated superior performance over traditional 2D metrics (such as simple animal speed). By capturing the full complexity of movement, these systems can more accurately identify the specific effects of drug-dose combinations in animal models, enabling more precise disease phenotyping.

Imitation Learning

3D data is also bridging the gap between biology and robotics. By using motion capture data as a constraint, reinforcement learning agents can now learn to mimic realistic gait patterns in physics simulators. This results in "virtual animals" that move with biological authenticity, providing a platform for testing behavioral hypotheses in silico.

Challenges and Future Directions

Despite these advances, several hurdles remain. Tracking animals in "naturalistic," cluttered environments is still difficult due to frequent occlusions. The burden of manual annotation for new species remains high, and models often struggle with "domain adaptation" when moved from controlled laboratory settings to the wild.

Future research is expected to integrate eye tracking and facial expression analysis into 3D skeletal models. This holistic approach will allow researchers to correlate physical movement with internal states, providing a truly comprehensive view of animal behavior and the neural logic that drives it.

Summary updated

Abstract

Animals move in three dimensions (3D). Thus, 3D measurement is necessary to report the true kinematics of animal movement. Existing 3D measurement techniques draw on specialized hardware, such as motion capture or depth cameras, as well as deep multi-view and monocular computer vision. Continued advances at the intersection of deep learning and computer vision will facilitate 3D tracking across more anatomical features, with less training data, in additional species, and within more natural, occlusive environments. 3D behavioral measurement enables unique applications in phenotyping, investigating the neural basis of behavior, and designing artificial agents capable of imitating animal behavior.
BibTeX
@inproceedings{marshall2022leaving,
  author = {Jesse D. Marshall and Tianqing Li and Joshua H. Wu and Timothy Dunn},
  title = {{Leaving Flatland: Advances in 3D behavioral measurement}},
  booktitle = {Current},
  year = {2022},
  doi = {10.1016/j.conb.2022.02.002},
}
Figure