Real-Time Recognition and Detection of Bactrocera minax (Diptera: Trypetidae) Grooming Behavior Using Body Region Localization and Improved C3D Network

Yong Sun, Wei Zhan, Tianyu Dong, Yuheng Guo, Hu Liu, Lianyou Gui, Zhiliang ZhangSensors 2023 · DOI

TL;DR

This paper introduces a novel method for automatically detecting and classifying the grooming behaviors of the Bactrocera minax fruit fly in real-time. By combining YOLOv3 for precise body region localization with an improved 3D Convolutional Neural Network (C3D), the system achieves high accuracy (93.46%) and significantly faster processing than manual observation, paving the way for more efficient pest management strategies.

What Problem Does This Paper Address?

Accurate identification and quantification of insect behavior are crucial for understanding pest ecology, developing effective control strategies, and ultimately protecting crops. Traditional methods of observing and analyzing insect behavior, such as manual frame-by-frame analysis, are time-consuming, labor-intensive, and struggle to keep pace with the sheer volume of data generated by modern high-throughput recording systems. This is particularly challenging for small insects like Bactrocera minax, where subtle movements and rapid behavior changes can be difficult to discern and record consistently. Existing automated methods often struggle with the dynamic nature of insect movement, leading to inaccuracies. This paper aims to bridge this gap by developing a robust, real-time system for recognizing specific grooming behaviors of Bactrocera minax.

Methods

The core of this research lies in a two-pronged approach:

  1. Body Region Localization (YOLOv3): To overcome the challenges posed by the insect's movement and ensure consistent focus on the relevant parts of the insect, the authors employ the YOLOv3 object detection algorithm. YOLOv3 is used to accurately identify and draw a bounding box around the Bactrocera minax in each video frame. This step is critical for isolating the insect and preparing it for subsequent analysis, reducing the computational load and potential noise from background elements.
  2. Improved 3D Convolutional Neural Network (C3D): For behavior recognition, the authors adapt and improve the C3D network, a standard for action recognition in videos. The key modifications include:
  3. Multi-scale Fusion: Incorporating features from different scales to better capture both high-level semantic information and low-level spatial details, which is crucial for distinguishing subtle grooming behaviors.
  4. ROI Stacking: Instead of processing raw video frames, the system extracts Regions of Interest (ROIs) from consecutive frames (specifically, 16 frames) identified by YOLOv3. These ROIs are then stacked chronologically to form a 3D "video cube" that is fed into the C3D network. This temporal stacking allows the network to learn the spatiotemporal dynamics of grooming actions.
  5. ROI Adjustment: A novel ROI adjustment mechanism is introduced to maintain stability between consecutive frames, especially during grooming. If the distance between the center points of bounding boxes in adjacent frames is below a threshold, the bounding box is adjusted to maintain consistency, thus mitigating jitter and improving the reliability of the input data for the C3D.

Datasets

The study utilized a custom dataset of Bactrocera minax behavior videos.

  • Data Acquisition: Videos were recorded using a Sony FDR-AXP60 camera at 1920x1080 resolution and 25 fps, with each adult insect filmed for 60 minutes in a petri dish.
  • Behavior Categories: The dataset was annotated for eight behaviors: head grooming, foreleg grooming, fore-mid leg grooming, hind leg grooming, mid-hind leg grooming, wing grooming, resting, and walking. The first six are the target grooming behaviors, while resting and walking were grouped as non-grooming behaviors.
  • Dataset Size: The experimental dataset comprised 20 adult Bactrocera minax, totaling 1200 minutes of footage. This was processed into feature videos, with 29,253 behavioral intervals identified. The dataset was split into training (22,315 intervals) and testing (6938 intervals) sets.

Key Insights and Ideas

  • Body Region Localization is Key: The paper strongly emphasizes that accurately localizing the insect's body region before attempting behavior recognition is crucial. This step significantly reduces noise and improves the efficiency and accuracy of the subsequent deep learning model, especially for small, mobile subjects.
  • 3D CNNs for Temporal Dynamics: The choice of a 3D CNN is justified by the need to capture the temporal evolution of grooming actions, which 2D CNNs alone cannot adequately represent. The spatiotemporal nature of grooming requires analyzing sequences of frames.
  • Multi-scale Fusion for Fine-grained Recognition: The improvement to the C3D network through multi-scale fusion is vital for distinguishing between similar grooming behaviors that involve different body parts. This allows the model to leverage both detailed local features and broader contextual information.
  • Real-time Performance is Achievable: The combination of efficient YOLOv3 detection and an optimized C3D network allows for real-time analysis, a significant improvement over manual methods and many existing automated approaches.

Key Results

  • Overall Accuracy: The proposed method achieved an impressive overall recognition accuracy of 93.46% for Bactrocera minax grooming behaviors.
  • Comparison to Manual Observation: The average difference in behavior detection between the proposed method and manual observation was approximately 12%.
  • Speed Advantage: The system is 3-5 times faster than manual observation, making it suitable for real-time applications.
  • Robustness to Walking: The method effectively distinguishes grooming from walking behavior, a common challenge in insect behavior analysis.
  • Behavior-Specific Performance: While overall accuracy is high, the paper notes that mid-hind leg and wing grooming showed slightly lower performance, potentially due to very short durations or rapid behavior switching.

Limitations and Future Work

  • Short Duration Behaviors: The authors acknowledge that variances in detection accuracy often occurred with behaviors of very short durations, which can be easily overlooked by human eyes or rapidly switched between.
  • Dataset Size and Labeling: Future work could involve expanding the training set and refining manual labeling to further improve accuracy.
  • Network Optimization: Exploring deeper 3D CNN architectures (e.g., 3D ResNet) or hybrid approaches (e.g., combining frame-by-frame and skipping frame methods) could enhance feature extraction and prediction.
  • Real-time Detection Speed: While the current method is fast, further optimization for even higher detection speeds is a potential future direction.

Summary updated

BibTeX
@article{sun2023real,
  author = {Yong Sun and Wei Zhan and Tianyu Dong and Yuheng Guo and Hu Liu and Lianyou Gui and Zhiliang Zhang},
  title = {{Real-Time Recognition and Detection of Bactrocera minax (Diptera: Trypetidae) Grooming Behavior Using Body Region Localization and Improved C3D Network}},
  journal = {Sensors},
  year = {2023},
  doi = {10.3390/s23146442},
}
Figure