Abstract
ENIGMA-51 is a novel egocentric video dataset designed to advance AI understanding of human behavior in complex industrial environments. Featuring 51 densely annotated sequences of technicians performing electrical repairs, the dataset provides a challenging benchmark for human-object interaction (HOI), action anticipation, and natural language intent recognition. By bridging the gap between "toy" assembly tasks and real-world industrial maintenance, ENIGMA-51 offers a robust foundation for developing intelligent industrial assistants.
Introduction
While computer vision has achieved significant success in daily activity recognition (e.g., EPIC-KITCHENS, Ego4D), the industrial sector remains underserved. Real-world factory data is difficult to acquire due to privacy, safety, and corporate secrecy. Consequently, existing industrial datasets often rely on simplified tasks like LEGO assembly, which fail to capture the visual complexity and high-stakes nature of professional maintenance. ENIGMA-51 addresses this by simulating a professional industrial laboratory where technicians repair high- and low-voltage electrical boards.
The ENIGMA-51 Dataset
The dataset consists of 51 high-resolution videos totaling approximately 22 hours of footage, capturing 14,036 distinct interactions. Recorded using Microsoft HoloLens 2, the sequences involve 19 subjects performing standardized repair procedures. To ensure robustness, the researchers introduced variability in tools (manual vs. electric screwdrivers) and components. Unlike datasets such as Assembly101 or MECCANO, ENIGMA-51 utilizes professional-grade equipment like oscilloscopes and power supplies, making it a more specialized and realistic resource for industrial AI.
Annotation Methodology
A key contribution of ENIGMA-51 is the depth and variety of its annotations. The researchers provided:
- Temporal & Verb Labels: Identification of "interaction key frames" using a specific taxonomy: first-contact, de-contact, take, and release.
- Object Bounding Boxes: Over 275,000 manually labeled objects across 25 classes.
- Hand States: Tracking of contact states for both left and right hands.
- Natural Language: 265 real-world worker utterances, augmented by synthetic data for intent recognition training.
- 3D Resources: Scans of the laboratory environment and industrial objects to support future synthetic data generation.
Benchmarking and Results
The authors established performance baselines using state-of-the-art models like ActionFormer and StillFast. The results highlight the significant difficulty of the industrial domain:
- Temporal Detection: Models achieved a mean Average Precision (mp-mAP) of 35.93%, indicating that pinpointing the exact boundaries of industrial tasks remains a challenge.
- HOI Detection: Detecting the triplet of
<hand, contact state, object>reached an mAP of 46.24%. - Action Anticipation: Predicting the next object and the "Time to Contact" (TTC) proved the most difficult task, with a Top-5 mAP of only 27.83%.
- Natural Language Understanding (NLU): Intent classification reached 86.7% accuracy, though models performed best when trained on real human utterances rather than purely synthetic text.
Key Insights
A critical finding is the importance of the "first-contact" distinction. In industrial safety, knowing exactly when a hand touches a potentially live circuit is often more vital than knowing when a tool is fully grasped. Furthermore, the visual similarity of electronic components makes "active object" detection significantly harder than in domestic settings, where objects are more visually distinct.
Conclusion and Future Work
ENIGMA-51 provides a specialized benchmark that reveals the limitations of current models in high-complexity industrial scenarios. Future research will focus on scaling the dataset through synthetic data generation using the provided 3D models, improving multi-modal fusion of audio and video, and developing safety-first AI models capable of detecting "near-miss" violations in real-time.
Summary updated