Introduction
In the fields of cognitive neuroscience and psychology, eye-tracking is a vital tool for understanding how humans process social information in naturalistic environments. However, analyzing this data requires "semantic segmentation"—the pixel-level labeling of objects of interest, such as specific human body parts. Manually labeling every frame of a feature-length film or a complex social scene is a prohibitive task. To address this, researchers at the National Institute of Mental Health (NIMH) have developed an automated deep learning tool capable of segmenting 11 distinct human body parts in video frames.
Methodology
The tool utilizes a Bayesian SegNet architecture, an encoder-decoder framework optimized for pixel-wise classification. A key innovation in this implementation is the use of concrete dropout at the center and output layers. Unlike traditional dropout methods with fixed probabilities, concrete dropout allows the network to dynamically tune its dropout rate during training. This enhances regularization and provides a potential measure of model uncertainty.
Developed using PyTorch, the model employs extensive data augmentation via the torchvision package—including random flipping and shifts in contrast, saturation, and hue—to ensure robustness across varying cinematic styles and lighting conditions.
Dataset and Training
The model was trained on the Pascal-Parts dataset, a benchmark that provides detailed part masks for the PASCAL VOC 2010 dataset. The researchers focused specifically on images containing "person" objects. To optimize performance, they simplified the classification task by merging lateral labels (e.g., combining "left arm" and "right arm" into a single "arm" category). The final model predicts 11 classes: hair, head, ear, eyebrows, legs, arms, mouth, neck, nose, torso, and background.
Key Results
The model's performance was quantitatively assessed using the Dice score, which measures the overlap between predicted masks and ground truth.
- High Performance: The model was most effective at identifying the Background (0.95), Eyes (0.62), and Mouth (0.62).
- Moderate Performance: The Head (0.60) and Eyebrows (0.60) also showed reliable segmentation.
- Challenges: The lowest performance was observed for Legs (0.38), likely due to high variability in posture, clothing, and frequent occlusion by environmental objects.
Qualitatively, the tool demonstrates a strong ability to generalize to "natural scenes" outside of its training set, successfully segmenting frames from various movies as shown in the figure above.
Discussion and Limitations
While the tool significantly reduces the manual workload for researchers, it has known limitations. Segmentation quality decreases during significant occlusion (e.g., an arm crossing the torso) and is sensitive to specific viewpoints, particularly direct frontal views that can blur the boundaries of facial features.
The researchers emphasize that for eye-tracking analysis, pixel-perfect accuracy in every frame is often unnecessary. Because these studies typically aggregate fixations over many frames, a "good enough" automated tool serves as a robust estimator for large-scale data analysis.
Conclusion
By releasing the full code and pre-trained weights, the authors provide an accessible resource for behavioral scientists. This open-science approach enables researchers without deep computer vision expertise to leverage advanced segmentation techniques in their own studies.
Code Availability: https://github.com/nih-fmrif/MLT_Body_Part_Segmentation
Summary updated