Social Behavioral Phenotyping of Drosophila With a 2D-3D Hybrid CNN Framework

Ziping Jiang, Paul L. Chazot, M. Emre Celebi, Danny Crookes, Richard JiangIEEE Access 2019 · OpenReview

TL;DR

This paper introduces a novel 2D+3D hybrid Convolutional Neural Network (CNN) framework for automatically analyzing the complex social behaviors of fruit flies (Drosophila) from video. By combining frame-level feature extraction with spatiotemporal analysis, the method achieves state-of-the-art performance in detecting and classifying subtle social interactions, paving the way for more efficient and objective biological research.

What Problem Does This Paper Address?

Understanding the social behaviors of model organisms like fruit flies is crucial for biological and medical research, helping scientists decipher the impact of genetics, diseases, and environmental factors on behavior. However, manually analyzing these behaviors from video is incredibly time-consuming, labor-intensive, and prone to human error and subjectivity. Existing automated methods often struggle with the fast-paced, subtle, and easily confusable social interactions of fruit flies, especially in low-resolution videos. This paper aims to bridge this gap by developing a robust deep learning framework capable of accurately detecting and localizing these complex behaviors.

Methods

The core of the proposed solution is a 2D+3D hybrid CNN framework. Here's a breakdown of the key components:

  • Base Layers (2D CNN): For each frame in a video clip, a 2D CNN (specifically, modified VGG or ResNet architectures) is used to extract spatial features. This approach is chosen to avoid losing crucial spatial information, which can be a problem with purely 3D CNNs when dealing with small, fast-moving subjects.
  • Spatiotemporal Feature Fusion (3D CNN): Features extracted from adjacent frames are then combined and fed into a 3D CNN. This allows the model to capture the temporal dynamics of the behaviors.
  • Spatiotemporal Region Proposal Network (SRPN): Inspired by object detection techniques like Faster R-CNN, an SRPN is employed to generate "tube proposals" that highlight potential regions of interest exhibiting a behavior across time.
  • Novel Anchor Labeling (ANK): To address the challenge of confusable behaviors, the authors introduce a new metric called ANK (Actionness-based Non-overlap Kernel) for labeling anchor boxes. This metric is more robust than traditional Intersection over Union (IoU) for distinguishing similar behaviors.
  • Super-Category Classifier: An optional super-category classifier is used to first distinguish between individual and social behaviors, which then helps the main classifier to better differentiate specific social interactions.
  • Sliding Window Approach: To manage computational costs, a sliding window is applied to the feature maps generated by the base layers, allowing for frame-wise predictions.

Datasets

The primary dataset used for training and evaluation is the Fly-vs-Fly dataset. This dataset contains approximately 22 hours of video footage of 47 pairs of fruit flies interacting. It is further divided into three subsets: "Boy meets boy" (for studying action sequences), "Aggression" (for quantifying genetic manipulation effects), and "Courtship" (for studying male courtship behavior). The authors specifically focused on the "Aggression" dataset for training and detecting three key behaviors: wing threat, hold, and tussle, due to their wide range and relatively balanced representation.

Key Insights and Ideas

  • 2D+3D Hybrid is Key: The combination of 2D CNNs for initial spatial feature extraction and 3D CNNs for spatiotemporal analysis proves more effective than purely 2D or 3D approaches for this task. This is particularly important for capturing subtle, fast movements without losing spatial detail.
  • ANK Outperforms IoU for Confusable Behaviors: The novel ANK metric for labeling anchor proposals significantly improves performance, especially for distinguishing between similar social behaviors, highlighting the limitations of standard IoU in such scenarios.
  • Bout Length Matters, But Not Always More is Better: While longer "bout lengths" (the number of frames considered for temporal context) generally improve performance, there's a point of diminishing returns. Very long bouts can introduce irrelevant information and lead to confusion, particularly for short-duration behaviors.
  • Super-Category Classifier: A Double-Edged Sword: The super-category classifier can boost training but also introduces instability, suggesting a trade-off between improved accuracy and training robustness.

Key Results

  • The proposed 2D+3D hybrid CNN framework achieved an overall mAP of 63.7% using the VGG16 base layers and a bout length of 5.
  • This performance is significantly higher than traditional object detection methods like Faster R-CNN (mAP 29.9%) and YOLO (mAP 22.4%), and competitive with or superior to other action detection methods like C3D (mAP 56.8%).
  • The ANK measurement for anchor labeling resulted in a 1.7% higher mAP for the VGG5 model compared to using IoU.
  • Training time for the models ranged from approximately 0.29s/frame (k=1) to 0.116s/frame (k=7) for training and 0.045s/frame to 0.231s/frame for prediction, demonstrating reasonable efficiency.

Limitations and Future Work

  • Super-Category Classifier Instability: The authors note that the super-category classifier, while potentially boosting performance, can make the training process unstable.
  • Confusable Behaviors: Despite improvements, distinguishing between highly similar social behaviors remains a challenge.
  • Future Directions: The authors suggest further research into deeper 3D CNN architectures and exploring more advanced feature extraction techniques. They also highlight the importance of capturing as much original information from the video as possible for better performance. The potential for using this framework for other laboratory animals is also implied.

Summary updated

Abstract

Behavioural phenotyping of drosophila is an important means in biological and medical research to identify the genetic, pathologic, or psychological impact on animal behavior. Automated behavioral phenotyping from videos has been a desired capability that can waive long-time boring manual work in behavioral analysis. In this paper, we introduced deep learning into this challenging topic and proposed a new 2D+3D hybrid CNN framework for drosophila's social behavioral phenotyping. In the proposed multi-task learning framework, action detection and localization of drosophila jointly are carried out with action classification, and a given video is divided into clips with fixed length. Each clip is fed into the system, and a 2-D CNN is applied to extract features at the frame level. Features extracted from adjacent frames are then connected and fed into a 3-D CNN with a spatial region proposal layer for classification. In such a 2D+3D hybrid framework, drosophila detection at the frame level enables the action analysis at different durations instead of a fixed period. We tested our framework with different base layers and classification architectures and validated the proposed 3D CNN-based social behavioral phenotyping framework under various models, detectors, and classifiers.
BibTeX
@inproceedings{jiang2019social,
  author = {Ziping Jiang and Paul L. Chazot and M. Emre Celebi and Danny Crookes and Richard Jiang},
  title = {{Social Behavioral Phenotyping of Drosophila With a 2D-3D Hybrid CNN Framework}},
  booktitle = {IEEE Access},
  year = {2019},
  url = {https://openreview.net/forum?id=Gyt5yeRp2w},
}
Figure