Testing human-hand segmentation on in-distribution and out-of-distribution data in human–robot interactions using a deep ensemble model

Reza Jalayer, Yuxin Chen, Masoud Jalayer, Carlotta Orsenigo, Masayoshi Tomizuka

TL;DR

This paper tackles the critical challenge of accurately segmenting human hands in human-robot interactions, especially in industrial settings where real-world conditions often differ from training data. By using a deep ensemble model and a novel, diverse industrial dataset, they demonstrate that models trained on domain-specific data significantly outperform those trained on general datasets, and introduce an uncertainty quantification method to flag unreliable predictions.

What Problem Does This Paper Address?

As robots become more integrated into human workspaces, particularly in Industry 4.0 and 5.0, ensuring safety and enabling seamless interaction is paramount. Human hands are particularly vulnerable and crucial for interaction. While deep learning models have shown promise in hand segmentation, most research evaluates them on "in-distribution" (ID) data – data that closely resembles their training sets. This is a major limitation because real-world industrial environments present "out-of-distribution" (OOD) scenarios: cluttered backgrounds, varying numbers of hands, different hand coverings (like gloves), unusual gestures, and motion blur. Existing datasets often lack these complexities, leading to models that fail when deployed in practice. This paper aims to bridge this gap by rigorously testing hand segmentation models under both ID and OOD conditions, specifically within an industrial context, and by quantifying their uncertainty.

Methods

The core of this work is the evaluation of pre-trained deep learning models for hand segmentation. The authors employ a deep ensemble model (DE-Mix), which combines multiple base learners (specifically, UNet and RefineNet) trained independently. This ensemble approach is known to improve predictive accuracy and, crucially, allows for uncertainty quantification. They use predictive entropy as a metric to measure the model's confidence in its predictions. Higher entropy indicates lower confidence, flagging potentially unreliable segmentations. This uncertainty analysis is key to identifying when a model is likely to fail, especially in OOD scenarios.

Datasets

The study utilizes a combination of existing and newly collected data:

  • Training Datasets:
    • EgoHands [13] & Ego2Hands [14]: Egocentric, non-industrial datasets.
  • HADR [15] & HAGS [12]: Industrial datasets, with HADR being synthetic and HAGS real-world. HAGS is noted for including gloved hands.
  • Test Dataset: A custom dataset was collected from real-world human-robot interactions using an industrial robotic arm. This dataset is designed to be challenging and diverse, featuring:
  • Backgrounds: Simple and cluttered industrial environments.
  • Number of Hands: Up to four hands.
  • Hand Conditions: Gloved and ungloved hands.
  • Gestures: Rare gestures (e.g., finger crossing).
  • Motion Blur: To simulate fast-moving hands.
  • Camera Views: Both egocentric (head-mounted) and static side-view cameras were used.

Key Insights and Ideas

  • Domain-Specific Data is Crucial: Models trained on industrial datasets (HAGS, HADR) significantly outperform those trained on general egocentric datasets (EgoHands, Ego2Hands) when tested in industrial OOD scenarios. This highlights that context matters immensely for generalization.
  • OOD Scenarios are a Major Hurdle: Even the best models struggle with OOD conditions like rare gestures and motion blur, showing a significant drop in accuracy compared to ID scenarios.
  • Uncertainty Quantification is a Safety Net: Predictive entropy effectively flags unreliable segmentations, especially in OOD scenarios. Models that show high uncertainty for OOD data are more trustworthy than those that confidently produce incorrect results.
  • Cluttered Backgrounds Increase Uncertainty: Models are less confident in their predictions when faced with complex, cluttered industrial backgrounds, even for ID scenarios.

Key Results

  • Segmentation Accuracy (mIoU):
    • Models trained on HAGS and HADR achieved higher mIoU scores across various conditions compared to EgoHands and Ego2Hands. For instance, in ID scenarios with one operator, HAGS achieved an mIoU of 0.496, while EgoHands was at 0.382.
  • In OOD scenarios like "two operators with gloves and rare gestures," mIoU dropped significantly for all models, but HAGS-trained models still performed better (e.g., 0.229) than EgoHands (0.058).
  • Models trained on non-industrial datasets struggled severely with OOD conditions, often achieving mIoU below 0.1 for rare gestures.
  • Predictive Entropy (Ē, Ēn):
    • While overall image entropy (Ē) remained relatively stable, hand-specific entropy (Ēn) was higher for OOD conditions across models trained on industrial data, indicating appropriate uncertainty.
  • Models trained on non-industrial data often showed low uncertainty even for incorrect OOD predictions, making them deceptively confident.
  • Qualitative Results: Visual examples showed that industrial-trained models could better handle gloved hands and cluttered scenes, while non-industrial models often misclassified tools or background elements as hands.
  • Speed: The DE-Mix model achieved a segmentation speed of 38 frames per second, suggesting potential for real-time applications.

Limitations and Future Work

  • Limitations:
    • The study did not fine-tune models on the test dataset, which inherently limits the absolute mIoU scores but emphasizes generalization capabilities.
  • The test dataset, while diverse, focused on specific industrial scenarios and camera viewpoints.
  • Future Work:
    • Fine-tuning: Exploring fine-tuning pre-trained models on industrial datasets to further boost performance.
  • Dataset Expansion: Creating more diverse datasets that include a wider range of rare gestures, occlusions, and more complex multi-operator scenarios.
  • Multimodal Data: Incorporating depth information or other sensor data to improve robustness.
  • Advanced Architectures: Investigating Transformer-based models for better spatial understanding and generalization.
  • Lightweight Models: Developing more efficient models for deployment in resource-constrained industrial environments.

Summary updated

Abstract

Reliable detection and segmentation of human hands are critical for enhancing safety and facilitating advanced interactions in human–robot collaboration. Current research predominantly evaluates hand segmentation under in-distribution (ID) data, which reflects the training data of deep learning (DL) models. However, this approach fails to address out-of-distribution (OOD) scenarios that often arise in real-world human–robot interactions. In this work, we make three key contributions: first we assess the generalization of deep learning (DL) models for hand segmentation under both ID and OOD scenarios, utilizing a newly collected industrial dataset that captures a wide range of real-world conditions including simple and cluttered backgrounds with industrial tools, varying numbers of hands (0 to 4), gloves, rare gestures, and motion blur. Our second contribution is considering both egocentric and static viewpoints. We evaluated the models trained on four datasets, i.e. EgoHands, Ego2Hands (egocentric mobile camera), HADR, and HAGS (static fixed viewpoint) by testing them with both egocentric (head-mounted) and static cameras, enabling robustness evaluation from multiple points of view. Our third contribution is introducing an uncertainty analysis pipeline based on the predictive entropy of predicted hand pixels. This procedure enables flagging unreliable segmentation outputs by applying thresholds established in the validation phase. This enables automatic identification and filtering of untrustworthy predictions, significantly improving segmentation reliability in OOD scenarios. For segmentation, we used a deep ensemble model composed of UNet and RefineNet as base learners. Our experiments demonstrate that models trained on industrial datasets (HADR, HAGS) outperform those trained on non-industrial datasets, both in segmentation accuracy and in their ability to flag unreliable outputs via uncertainty estimation. These findings underscore the necessity of domain-specific training data and show that our uncertainty analysis pipeline can provide a practical safety layer for real-world deployment.
BibTeX
@misc{jalayer2025testing,
  author = {Reza Jalayer and Yuxin Chen and Masoud Jalayer and Carlotta Orsenigo and Masayoshi Tomizuka},
  title = {{Testing human-hand segmentation on in-distribution and out-of-distribution data in human–robot interactions using a deep ensemble model}},
  year = {2025},
  eprint = {2501.07713},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2501.07713},
}
Figure