DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila

Semih Günel, Helge Rhodin, Daniel Morales, João Campagnolo, Pavan P Ramdya, Pascal FuaeLife 2019 · DOI

TL;DR

Researchers have developed DeepFly3D, a novel deep learning system for precisely tracking the 3D movements of fruit flies. This tool overcomes previous limitations in small animal pose estimation, enabling more accurate behavioral analysis and a deeper understanding of neural circuits controlling movement.

What Problem Does This Paper Address?

Understanding how an animal's nervous system controls its movements requires precise tracking of its body parts in three-dimensional (3D) space. While deep learning has revolutionized 2D pose estimation for humans and even some lab animals, accurately reconstructing the 3D pose of tiny organisms like Drosophila melanogaster (fruit flies) has remained a significant challenge. Existing methods for 3D pose estimation often rely on markers that can impede movement in small animals, or they require complex, time-consuming calibration procedures that are difficult to implement for the fly's delicate anatomy. Furthermore, 2D pose data, while useful, can be ambiguous due to occlusions and perspective changes, limiting the depth of behavioral analysis.

Methods

DeepFly3D tackles this problem with a multi-pronged approach:

  • Deep Learning for 2D Pose Estimation: The system starts by using a state-of-the-art deep neural network (specifically, a modified Stacked Hourglass network) to detect 38 keypoints (joints and landmarks) on the fly in 2D images from multiple synchronized cameras.
  • Multi-View Geometry and Triangulation: By combining the 2D detections from seven synchronized cameras, the system triangulates these points to infer their 3D positions.
  • Novel Camera Calibration: Instead of relying on external calibration patterns, DeepFly3D ingeniously uses the fly itself as the calibration target. It leverages bundle adjustment and geometric constraints of the fly's anatomy to simultaneously estimate camera positions and orientations, as well as the 3D locations of the fly's joints.
  • Pictorial Structures for Error Correction: To refine the 3D pose estimates and correct for potential errors or ambiguities in the 2D detections, the system employs pictorial structures. This graphical model approach enforces geometric consistency and helps to eliminate spurious detections.
  • Self-Supervision and Active Learning: DeepFly3D incorporates a self-supervision strategy where the system uses its own multi-view geometry to identify and correct erroneous 2D predictions. This allows the network to iteratively improve its performance with minimal manual annotation, making the training process more efficient.

Datasets

The paper utilized a combination of existing and self-generated data:

  • MPII Human Pose Dataset: Used for pre-training the initial deep neural network, leveraging existing knowledge from human pose estimation.
  • Custom Drosophila Datasets: The authors collected and annotated extensive datasets of tethered, adult Drosophila from multiple camera views. This included data from various transgenic lines (MDN>CsChrimson, aDN>CsChrimson) and control animals, captured during specific behaviors like backward walking and grooming, often induced by optogenetic stimulation.
  • Human 3.6M Dataset: Used to demonstrate the adaptability of DeepFly3D for estimating 3D human pose, showcasing its flexibility beyond Drosophila.

Key Insights and Ideas

  • The Fly as a Calibration Tool: The most striking innovation is using the animal itself for camera calibration. This elegantly sidesteps the challenges of creating and aligning tiny, precise calibration targets for small organisms.
  • Multi-View Redundancy is Key: The system heavily relies on the redundancy provided by multiple camera views to achieve robustness and accuracy, allowing for error detection and correction that would be impossible with a single camera.
  • 3D Pose Unlocks Deeper Behavioral Insights: The paper convincingly demonstrates that 3D joint angles provide a more robust and less ambiguous representation of animal behavior for unsupervised analysis compared to 2D pose data, which can be heavily influenced by viewing angle.
  • Iterative Refinement is Powerful: The combination of deep learning, multi-view geometry, and pictorial structures, coupled with active learning, creates a powerful feedback loop for continuous improvement of pose estimation accuracy.

Key Results

  • High Accuracy: DeepFly3D achieves high accuracy in 3D pose estimation for tethered Drosophila, with a reported general accuracy of 98.2% before applying pictorial structures, and 99.2% after.
  • Robustness: The system is robust to occlusions and motion blur, thanks to the multi-view approach and pictorial structure corrections.
  • Efficiency: The software can run at high frame rates (up to 100 FPS for error correction), making it suitable for analyzing large behavioral datasets.
  • Improved Behavioral Analysis: Unsupervised behavioral classification using 3D joint angles from DeepFly3D showed clear segregation of behaviors (e.g., backward walking, grooming) that was not achievable with 2D pose data alone.

Limitations and Future Work

  • Tethered Animals: The current implementation is designed for tethered flies. While the authors suggest adaptability to other animals, extending it to freely moving animals would be a significant advancement.
  • Manual Annotation for Initialization: While the system uses active learning to reduce manual annotation, an initial set of manual annotations is still required to bootstrap the process.
  • Generalizability to Other Animals: While the framework is designed to be adaptable, specific modifications to parameters like bone segment lengths and camera focal distances would be necessary for different species. The authors suggest this as a future direction, demonstrating its potential for studying humans, primates, and other insects.
  • Real-time Free-Moving Animals: The authors mention the possibility of lifting 2D detections to 3D or directly regressing to 3D for monocular images, which could pave the way for real-time 3D pose estimation in freely moving animals.

Summary updated

Abstract

Studying how neural circuits orchestrate limbed behaviors requires the precise measurement of the positions of each appendage in three-dimensional (3D) space. Deep neural networks can estimate two-dimensional (2D) pose in freely behaving and tethered animals. However, the unique challenges associated with transforming these 2D measurements into reliable and precise 3D poses have not been addressed for small animals including the fly, <i>Drosophila melanogaster</i>. Here, we present DeepFly3D, a software that infers the 3D pose of tethered, adult <i>Drosophila</i> using multiple camera images. DeepFly3D does not require manual calibration, uses pictorial structures to automatically detect and correct pose estimation errors, and uses active learning to iteratively improve performance. We demonstrate more accurate unsupervised behavioral embedding using 3D joint angles rather than commonly used 2D pose data. Thus, DeepFly3D enables the automated acquisition of <i>Drosophila</i> behavioral measurements at an unprecedented level of detail for a variety of biological applications.
BibTeX
@inproceedings{gunel2019deepfly,
  author = {Semih Günel and Helge Rhodin and Daniel Morales and João Campagnolo and Pavan P Ramdya and Pascal Fua},
  title = {{DeepFly3D, a deep learning-based approach for 3D limb and appendage tracking in tethered, adult Drosophila}},
  booktitle = {eLife},
  year = {2019},
  doi = {10.7554/elife.48571},
}

Loading related content...

Figure