A Deep Neural Network Tool for Automatic Segmentation of Human Body Parts in Natural Scenes

Patrick McClure, Gabrielle Reimann, Michal Ramot, Francisco PereiraarXiv 2020 · arXiv

Introduction

In the fields of cognitive neuroscience and psychology, eye-tracking is a vital tool for understanding how humans process social information in naturalistic environments. However, analyzing this data requires "semantic segmentation"—the pixel-level labeling of objects of interest, such as specific human body parts. Manually labeling every frame of a feature-length film or a complex social scene is a prohibitive task. To address this, researchers at the National Institute of Mental Health (NIMH) have developed an automated deep learning tool capable of segmenting 11 distinct human body parts in video frames.

Methodology

The tool utilizes a Bayesian SegNet architecture, an encoder-decoder framework optimized for pixel-wise classification. A key innovation in this implementation is the use of concrete dropout at the center and output layers. Unlike traditional dropout methods with fixed probabilities, concrete dropout allows the network to dynamically tune its dropout rate during training. This enhances regularization and provides a potential measure of model uncertainty.

Developed using PyTorch, the model employs extensive data augmentation via the torchvision package—including random flipping and shifts in contrast, saturation, and hue—to ensure robustness across varying cinematic styles and lighting conditions.

Dataset and Training

The model was trained on the Pascal-Parts dataset, a benchmark that provides detailed part masks for the PASCAL VOC 2010 dataset. The researchers focused specifically on images containing "person" objects. To optimize performance, they simplified the classification task by merging lateral labels (e.g., combining "left arm" and "right arm" into a single "arm" category). The final model predicts 11 classes: hair, head, ear, eyebrows, legs, arms, mouth, neck, nose, torso, and background.

Key Results

The model's performance was quantitatively assessed using the Dice score, which measures the overlap between predicted masks and ground truth.

  • High Performance: The model was most effective at identifying the Background (0.95), Eyes (0.62), and Mouth (0.62).
  • Moderate Performance: The Head (0.60) and Eyebrows (0.60) also showed reliable segmentation.
  • Challenges: The lowest performance was observed for Legs (0.38), likely due to high variability in posture, clothing, and frequent occlusion by environmental objects.
Qualitative results showing original video frames and their corresponding automated segmentations across various natural scenes.

Qualitatively, the tool demonstrates a strong ability to generalize to "natural scenes" outside of its training set, successfully segmenting frames from various movies as shown in the figure above.

Discussion and Limitations

While the tool significantly reduces the manual workload for researchers, it has known limitations. Segmentation quality decreases during significant occlusion (e.g., an arm crossing the torso) and is sensitive to specific viewpoints, particularly direct frontal views that can blur the boundaries of facial features.

The researchers emphasize that for eye-tracking analysis, pixel-perfect accuracy in every frame is often unnecessary. Because these studies typically aggregate fixations over many frames, a "good enough" automated tool serves as a robust estimator for large-scale data analysis.

Conclusion

By releasing the full code and pre-trained weights, the authors provide an accessible resource for behavioral scientists. This open-science approach enables researchers without deep computer vision expertise to leverage advanced segmentation techniques in their own studies.

Code Availability: https://github.com/nih-fmrif/MLT_Body_Part_Segmentation

Summary updated

Abstract

This short article describes a deep neural network trained to perform automatic segmentation of human body parts in natural scenes. More specifically, we trained a Bayesian SegNet with concrete dropout on the Pascal-Parts dataset to predict whether each pixel in a given frame was part of a person's hair, head, ear, eyebrows, legs, arms, mouth, neck, nose, or torso.
BibTeX
@misc{mcclure2020deep,
  author = {Patrick McClure and Gabrielle Reimann and Michal Ramot and Francisco Pereira},
  title = {{A Deep Neural Network Tool for Automatic Segmentation of Human Body Parts in Natural Scenes}},
  year = {2020},
  eprint = {2009.09900},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2009.09900},
}
Figure