MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain

Francesco Ragusa, Antonino Furnari, Giovanni Maria Farinella

Abstract

MECCANO is a novel multimodal egocentric dataset designed to facilitate research into human behavior understanding within industrial-like environments. By capturing synchronized RGB, depth, and gaze data as subjects assemble a complex technical toy motorbike, the researchers provide a rigorous benchmark for tasks such as action recognition, active object detection, and action anticipation. This paper presents the dataset's composition and evaluates baseline performances, highlighting the critical role of depth information in procedural task analysis.

Introduction

While computer vision has achieved significant success in third-person human behavior understanding, egocentric (first-person) vision remains a formidable challenge. Most existing egocentric datasets are situated in domestic environments, such as kitchens or living rooms. However, industrial settings introduce unique complexities: actions are strictly procedural, objects are often minute (e.g., screws, specialized tools), and the consequences of errors are substantial.

Collecting data in actual industrial facilities is often prohibited by privacy regulations and corporate confidentiality. To bridge this gap, the MECCANO dataset serves as a high-quality "industrial-like" proxy. It enables the development of AI assistants capable of monitoring production pipelines, enhancing worker safety, and providing real-time guidance.

The MECCANO Dataset

The primary contribution of this work is the MECCANO Multimodal Dataset. It captures 20 subjects across two countries (Italy and the UK) as they assemble a technical toy motorbike consisting of numerous small, fine-grained components.

The MECCANO dataset includes a wide variety of small components, tools, and a procedural assembly booklet.

The dataset comprises approximately 300,000 frames across 20 videos, featuring 61 action classes and 20 object classes. Annotations are extensive, including temporal action segments, spatial bounding boxes for hands and active objects, and labels for "next-active" objects. Unlike broader datasets like Ego4D, MECCANO is specifically tailored for procedural, industrial-style tasks and includes synchronized gaze and depth data.

Methodology

Data was collected using a custom wearable headset equipped with an Intel RealSense SR300 for RGB and depth capture, and a Pupil Core eye tracker for gaze tracking. The researchers established baselines by evaluating state-of-the-art models on five key tasks:

  1. Action Recognition: Identifying the current action.
  2. Active Object Detection: Identifying objects currently being manipulated.
  3. Egocentric Human-Object Interaction (EHOI) Detection: Identifying <hand, verb, object> triplets.
  4. Action Anticipation: Predicting the subsequent action.
  5. Next-Active Object Detection: Predicting the next object the user will interact with.

The evaluation utilized the SlowFast architecture for video analysis and Faster R-CNN for object detection, modified to integrate depth and gaze modalities.

Results and Insights

The evaluation revealed several key findings regarding multimodal learning in procedural tasks:

  • The Impact of Depth: Integrating depth maps with standard RGB video provided a significant performance boost in action recognition and interaction detection. Depth information helps models distinguish between a hand merely moving near an object and a hand actively grasping it.
  • Small Object Complexity: Recognizing small, similar-looking industrial components (e.g., different types of bars or screws) proved much more difficult than distinguishing common household items. The baseline model achieved a mean Average Precision (mAP) of 30.39% for active object recognition.
  • The Anticipation Challenge: Predicting future actions and "next-active" objects remains significantly harder than recognizing current ones. While action recognition reached a Top-1 Accuracy of 49.66% using multimodal data, anticipation tasks showed lower performance, indicating a need for models that understand long-term assembly sequences.
Synchronized RGB and depth frames illustrate the multimodal nature of the dataset across different subjects and assembly stages.
  • Gaze as a Signal: While gaze data improved results, the improvement was less pronounced than depth. This suggests that current neural architectures may not yet be fully optimized to leverage human visual attention effectively.

Conclusion

MECCANO represents a significant step toward understanding human behavior in industrial contexts. While it serves as a proxy for real-world factories, its dense annotations and multimodal nature provide a challenging testbed for AI. Future research should focus on developing architectures that more effectively fuse gaze and depth signals and on creating long-term anticipation models capable of understanding multi-step assembly processes.

Summary updated

BibTeX
@article{ragusa2023meccano,
  author = {Francesco Ragusa and Antonino Furnari and Giovanni Maria Farinella},
  title = {{MECCANO: A Multimodal Egocentric Dataset for Humans Behavior Understanding in the Industrial-like Domain}},
  journal = {Comput. Vis. Image Underst.},
  year = {2023},
  doi = {10.1016/j.cviu.2023.103764},
}
Figure