ENIGMA-51: Towards a Fine-Grained Understanding of Human Behavior in Industrial Scenarios

Francesco Ragusa, Rosario Leonardi, Michele Mazzamuto, Claudia Bonanno, Rosario Scavo, Antonino Furnari, Giovanni Maria Farinella

Abstract

ENIGMA-51 is a novel egocentric video dataset designed to advance AI understanding of human behavior in complex industrial environments. Featuring 51 densely annotated sequences of technicians performing electrical repairs, the dataset provides a challenging benchmark for human-object interaction (HOI), action anticipation, and natural language intent recognition. By bridging the gap between "toy" assembly tasks and real-world industrial maintenance, ENIGMA-51 offers a robust foundation for developing intelligent industrial assistants.

Introduction

While computer vision has achieved significant success in daily activity recognition (e.g., EPIC-KITCHENS, Ego4D), the industrial sector remains underserved. Real-world factory data is difficult to acquire due to privacy, safety, and corporate secrecy. Consequently, existing industrial datasets often rely on simplified tasks like LEGO assembly, which fail to capture the visual complexity and high-stakes nature of professional maintenance. ENIGMA-51 addresses this by simulating a professional industrial laboratory where technicians repair high- and low-voltage electrical boards.

The ENIGMA-51 Dataset

The dataset consists of 51 high-resolution videos totaling approximately 22 hours of footage, capturing 14,036 distinct interactions. Recorded using Microsoft HoloLens 2, the sequences involve 19 subjects performing standardized repair procedures. To ensure robustness, the researchers introduced variability in tools (manual vs. electric screwdrivers) and components. Unlike datasets such as Assembly101 or MECCANO, ENIGMA-51 utilizes professional-grade equipment like oscilloscopes and power supplies, making it a more specialized and realistic resource for industrial AI.

Annotation Methodology

A key contribution of ENIGMA-51 is the depth and variety of its annotations. The researchers provided:

  • Temporal & Verb Labels: Identification of "interaction key frames" using a specific taxonomy: first-contact, de-contact, take, and release.
  • Object Bounding Boxes: Over 275,000 manually labeled objects across 25 classes.
  • Hand States: Tracking of contact states for both left and right hands.
  • Natural Language: 265 real-world worker utterances, augmented by synthetic data for intent recognition training.
  • 3D Resources: Scans of the laboratory environment and industrial objects to support future synthetic data generation.
Comprehensive overview of the ENIGMA-51 dataset, showcasing 2D/3D annotations, hand-object interaction boxes, and procedural text.

Benchmarking and Results

The authors established performance baselines using state-of-the-art models like ActionFormer and StillFast. The results highlight the significant difficulty of the industrial domain:

  • Temporal Detection: Models achieved a mean Average Precision (mp-mAP) of 35.93%, indicating that pinpointing the exact boundaries of industrial tasks remains a challenge.
  • HOI Detection: Detecting the triplet of <hand, contact state, object> reached an mAP of 46.24%.
  • Action Anticipation: Predicting the next object and the "Time to Contact" (TTC) proved the most difficult task, with a Top-5 mAP of only 27.83%.
  • Natural Language Understanding (NLU): Intent classification reached 86.7% accuracy, though models performed best when trained on real human utterances rather than purely synthetic text.
Sequence demonstrating the 'Time to Contact' (TTC) metric and the identification of the 'first-contact' key frame.

Key Insights

A critical finding is the importance of the "first-contact" distinction. In industrial safety, knowing exactly when a hand touches a potentially live circuit is often more vital than knowing when a tool is fully grasped. Furthermore, the visual similarity of electronic components makes "active object" detection significantly harder than in domestic settings, where objects are more visually distinct.

Conclusion and Future Work

ENIGMA-51 provides a specialized benchmark that reveals the limitations of current models in high-complexity industrial scenarios. Future research will focus on scaling the dataset through synthetic data generation using the provided 3D models, improving multi-modal fusion of audio and video, and developing safety-first AI models capable of detecting "near-miss" violations in real-time.

Summary updated

Abstract

ENIGMA-51 is a new egocentric dataset acquired in an industrial scenario by 19 subjects who followed instructions to complete the repair of electrical boards using industrial tools (e.g., electric screwdriver) and equipments (e.g., oscilloscope). The 51 egocentric video sequences are densely annotated with a rich set of labels that enable the systematic study of human behavior in the industrial domain. We provide benchmarks on four tasks related to human behavior: 1) untrimmed temporal detection of human-object interactions, 2) egocentric human-object interaction detection, 3) short-term object interaction anticipation and 4) natural language understanding of intents and entities. Baseline results show that the ENIGMA-51 dataset poses a challenging benchmark to study human behavior in industrial scenarios. We publicly release the dataset at https://iplab.dmi.unict.it/ENIGMA-51.
BibTeX
@inproceedings{ragusa2024enigma,
  author = {Francesco Ragusa and Rosario Leonardi and Michele Mazzamuto and Claudia Bonanno and Rosario Scavo and Antonino Furnari and Giovanni Maria Farinella},
  title = {{ENIGMA-51: Towards a Fine-Grained Understanding of Human Behavior in Industrial Scenarios}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
  doi = {10.1109/wacv57701.2024.00449},
}
Figure