FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving Drosophila

Virginie Uhlmann, Pavan P Ramdya, Ricard Delgado-Gonzalo, Richard Benton, Michaël UnserbioRxiv 2016 · DOI

TL;DR

This paper introduces FlyLimbTracker, an open-source software tool that uses active contour models (snakes) to semi-automatically track the body and leg segments of freely behaving fruit flies. This method overcomes limitations of previous approaches, enabling more precise and high-throughput analysis of complex fly behaviors, crucial for understanding the neural basis of movement.

What Problem Does This Paper Address?

Understanding how brains control movement requires precise, quantitative, and high-throughput measurement of animal behavior. The fruit fly, Drosophila melanogaster, is an excellent model organism due to its genetic tractability and relatively simple nervous system. However, accurately tracking the fine movements of individual leg segments in freely behaving flies has been a significant challenge. Existing methods often require complex experimental setups, manual marker placement, or provide only limited information about leg movements. This gap hinders detailed investigations into the neural mechanisms underlying behaviors like walking, grooming, and courtship.

Methods

FlyLimbTracker employs an active contour model, commonly known as snakes, to segment and track the fly's body and legs. The process involves:

  1. Semi-automated Initialization: The user manually outlines the fly's body and leg tips in an initial video frame.
  2. Body Segmentation: A closed snake model, parameterized by 18 control points, is optimized to fit the fly's body contour. This model uses a combination of edge-based energy (detecting gradients), region-based energy (maximizing contrast between the fly and background), and shape-prior energy (maintaining a characteristic fly shape).
  3. Leg Segmentation: Open snake models, each with 4 control points, are used for individual legs. These are initialized by first using dynamic programming to find an optimal path from a body anchor to the user-defined leg tip, and then transforming this path into a parametric curve for snake optimization.
  4. Automated Tracking (Propagation): For subsequent frames, the software propagates the segmented body and leg models. Body tracking is achieved by optimizing the previous frame's snake model. Leg tracking is more sophisticated, using a combination of energies that consider ridge detection, distance transforms, segment length consistency, and tip candidate locations.
  5. User Interaction: The system allows users to interrupt tracking to correct errors, and the corrected annotations are then used to resume automated tracking.

The software is implemented as a plugin for the Icy image processing platform, making it accessible and extensible.

Datasets

The paper describes experiments using adult female Drosophila melanogaster of the Canton-S strain. Behavioral videos were captured at high speed (236 frames-per-second) and high resolution (2560 x 918 pixels). The authors systematically varied spatial and temporal resolutions of these videos to test the algorithm's robustness. Five common Drosophila behaviors were analyzed: walking straight, turning, foreleg grooming, head grooming, and abdominal grooming. No new datasets were introduced; rather, existing video data was manipulated to assess performance under different conditions.

Key Insights and Ideas

  • Active Contours for Biological Motion: The paper demonstrates the effectiveness of active contour models (snakes) for precise, semi-automated tracking of complex biological structures like insect limbs, even in freely moving animals.
  • Hybrid Approach for Robustness: Combining dynamic programming for initial leg pathfinding with snake models for optimization provides a robust solution that leverages the strengths of both discrete and continuous methods.
  • Generalizability: The method's design, particularly its reliance on computational image processing rather than specialized hardware, suggests it can be adapted for tracking leg movements in other species with modifications to the snake models.
  • Trade-off Between Automation and Manual Correction: The study highlights how video resolution impacts the balance between automated processing time and the need for user intervention, offering insights into optimizing data acquisition and analysis pipelines.

Key Results

  • FlyLimbTracker successfully tracks body and leg segments in unmarked, freely behaving Drosophila.
  • The approach is robust to significant variations in spatial (up to 8x down-sampling) and temporal (down to 14.75 fps) resolutions.
  • While automated body tracking required minimal user intervention, leg tracking showed increased need for manual corrections at lower resolutions.
  • Annotation time per frame was significantly longer for walking and turning behaviors at lower resolutions, emphasizing the importance of temporal overlap for automated tracking.
  • The software provides multiple visualization options for analyzing leg segment trajectories and speeds, enabling detailed characterization of various behaviors.

Limitations and Future Work

  • Body Overlap Constraint: Tracking currently requires overlap of the fly's body between successive frames, setting a lower bound on video temporal resolution. The authors suggest using algorithms like the Hungarian algorithm for nearest-neighbor matching to improve this.
  • Leg Model Detail: The authors propose adding more control points to FlyLimbTracker to more precisely annotate segments like the thorax-coxa-trochanter.
  • Semi-Automation: The requirement for user initialization makes the current system semi-automated. Future work could involve further automation to enable high-throughput batch processing of multiple videos.
  • Species Adaptability: While generalizable, adapting the snake models for other species (e.g., mice, stick insects) would require modifications to the model's shape.

Summary updated

Abstract

Abstract Understanding the biological underpinnings of movement and action requires the development of tools for precise, quantitative, and high-throughput measurements of animal behavior. Drosophila melanogaster provides an ideal model for developing such tools: the fly has unparalleled genetic accessibility and depends on a relatively compact nervous system to generate sophisticated limbed behaviors including walking, reaching, grooming, courtship, and boxing. Here we describe a method that uses active contours to semi-automatically track body and leg segments from video image sequences of unmarked, freely behaving Drosophila . We show that this approach is robust to wide variations in video spatial and temporal resolution and that it can be used to measure leg segment motions during a variety of locomotor and grooming behaviors. FlyLimbTracker, the software implementation of this method, is open-source and our approach is generalizable. This opens up the possibility of tracking leg movements in other species by modifications of underlying active contour models. Author Summary In terrestrial animals, including humans, fundamental actions like locomotion and grooming emerge from the displacement of multiple limbs through space. Therefore, precise measurements of limb movements are critical for investigating and, ultimately, understanding the neural basis for behavior. The vinegar fly, Drosophila melanogaster , is an attractive animal model for uncovering general principles about limb control since its genome and nervous system are easy to manipulate. However, existing methods for measuring leg movements in freely behaving Drosophila have significant drawbacks: they require complicated experimental setups and provide limited information about each leg. Here we report a new method - and provide its open-source software implementation, FlyLimbTracker - for tracking the body and leg segments of freely behaving flies using only computational image processing approaches. We illustrate the power of this method by tracking fly limbs during five distinct walking and grooming behaviors and from videos across a wide range of spatial and temporal resolutions. Our approach is generalizable, allowing researchers to use and customize our software for limb tracking in Drosophila and in other species.
BibTeX
@misc{uhlmann2016flylimbtracker,
  author = {Virginie Uhlmann and Pavan P Ramdya and Ricard Delgado-Gonzalo and Richard Benton and Michaël Unser},
  title = {{FlyLimbTracker: an active contour based approach for leg segment tracking in unmarked, freely behaving <i>Drosophila</i>}},
  year = {2016},
  doi = {10.1101/089714},
  howpublished = {bioRxiv},
}
Figure