IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting

Tim J. Schoonbeek, Tim Houben, Hans Onvlee, Peter H.n. de With, Fons van der SommenWACV 2024

TL;DR

This paper introduces "IndustReal," a novel egocentric video dataset and the task of Procedure Step Recognition (PSR) to address the critical need for understanding procedural tasks in industrial settings. Unlike prior work, IndustReal captures a wide variety of execution errors and allows for flexible step ordering, enabling more robust and realistic procedural understanding systems.

What Problem Does This Paper Address?

Current approaches to understanding procedural tasks in videos, like action recognition (AR) and assembly state detection (ASD), fall short for industrial applications. AR often only identifies what action is being performed, not if it's completed correctly, while ASD struggles with the combinatorial explosion of states in complex procedures and doesn't inherently leverage procedural knowledge. A key limitation is the lack of focus on correct step completion and the handling of errors, which are common and costly in real-world industrial scenarios. Imagine a technician missing a crucial step or using the wrong part – current systems might not flag this, leading to significant rework. This paper aims to bridge this gap by introducing PSR, which focuses on recognizing the correct completion and order of procedural steps, even in the presence of errors.

Methods

The core contribution is the definition of Procedure Step Recognition (PSR). This task aims to extract an estimate of all procedure steps correctly performed up to a given time t, based on sensory inputs and a descriptive set of procedural actions. The key innovation is the explicit leveraging of procedural knowledge (e.g., expected next steps) to constrain possibilities and the focus on recognizing completed steps rather than just ongoing actions.

The paper also proposes three evaluation metrics for PSR:

  1. Procedure Order Similarity (POS): Measures how well the predicted sequence of completed steps matches the ground truth sequence, using a modified Damerau-Levenshtein distance to penalize transpositions less and avoid favoring models with many false positives.
  2. F1 Score: Evaluates true positives, false positives, and false negatives based on whether a step was predicted correctly in relation to its actual completion time and presence in the ground truth.
  3. Average Delay (τ): Quantifies the temporal difference between the ground-truth completion of a step and its recognition, excluding false positives and negatives.

Datasets

The paper introduces the IndustReal dataset, a new benchmark for procedural understanding.

  • Content: Egocentric videos of 27 participants performing assembly and maintenance tasks on a construction-toy car (STEMFIE).
  • Modalities: Includes stereo RGB, visible light, depth, IR, gaze, and hand joint tracking.
  • Key Features:
    • Variety of Errors: Features 38 errors, including omissions and execution errors (e.g., wrong part used), with many errors unique to validation/test sets to test robustness.
  • Subgoal-Oriented Execution: Participants have flexibility in the order of subtasks, mimicking industrial procedures.
  • Open-Source Geometries: 3D models of all parts are publicly available to facilitate synthetic data generation and sim2real transfer.
  • 3D Printed Parts: Ensures reproducibility and future availability.
  • Annotations: Provides labels for AR, ASD, and the new PSR task.

The paper also references existing datasets like MECCANO, Assembly101, and BRIO-TA for comparison.

Key Insights and Ideas

  • The Importance of "Completion": The paper strongly argues that knowing if a step is completed correctly is far more valuable in industrial settings than simply knowing what action is being performed. This shifts the focus from action recognition to a more task-oriented understanding.
  • Errors are Crucial: Industrial procedures are rife with errors. Datasets that don't explicitly include and annotate these errors will lead to models that fail in real-world deployment. IndustReal's inclusion of novel error types is a significant step.
  • Procedural Knowledge is Key: Simply relying on visual cues is insufficient. Incorporating knowledge about the expected sequence of actions can drastically improve performance and reduce ambiguity, especially when visual differences between states are subtle.
  • Synthetic Data for Generalization: The public availability of 3D models is a strategic choice to enable researchers to leverage synthetic data for training and sim2real adaptation, a critical factor for scaling to diverse industrial tasks.

Key Results

  • AR Benchmark: The MViTv2 transformer outperformed SlowFast on the AR task. Ensembles of modalities (RGB, VL, stereo) showed superior performance over individual modalities, highlighting complementary information.
  • ASD Benchmark: Combining synthetic and real-world data for training YOLOv8-m yielded the highest performance on the ASD task. However, a significant performance drop was observed when evaluating on entire videos compared to frame-level annotations, indicating challenges with fine-grained visual differences and error states.
  • PSR Benchmark: Baseline PSR models showed promising results on error-free recordings but struggled significantly with out-of-distribution execution errors. The best performing baseline achieved a POS of 0.797 and F1 of 0.883 on error-free data. The use of procedural information (as in baseline B3) significantly improved performance over models that solely relied on ASD output.

Limitations and Future Work

  • PSR Baseline Performance on Errors: The current PSR baselines, while showing promise, fail to generalize well to execution errors. This is a major limitation for real-world industrial applicability.
  • Temporal Component: While the average delay metric is introduced, the paper notes that comparing only POS and F1 score lacks a temporal component.
  • Future Work: The authors plan to focus on improving PSR baseline performance and scalability, particularly in handling execution errors. They aim to make PSR a more viable and attractive solution for industrial use cases. Further research into leveraging procedural knowledge more effectively and exploring more sophisticated error detection and correction mechanisms is implied.

Summary updated

BibTeX
@inproceedings{schoonbeek2024industreal,
  author = {Tim J. Schoonbeek and Tim Houben and Hans Onvlee and Peter H.N. de With and Fons van der Sommen},
  title = {{IndustReal: A Dataset for Procedure Step Recognition Handling Execution Errors in Egocentric Videos in an Industrial-Like Setting}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2024},
}
Figure