TL;DR
Researchers have developed a way to detect mistakes in skilled activities (like assembling a tent or repairing machinery) using only eye-gaze data from egocentric videos. By training a model to "complete" a partial gaze trajectory, they can identify errors when a user's actual gaze deviates from the predicted "normal" pattern—all without needing a single manual mistake label.
What Problem Does This Paper Address?
Detecting when someone makes a mistake while performing a complex task is a holy grail for Augmented Reality (AR) assistants. If a wearable device knows you've missed a step or are confused, it can proactively offer help.
Traditionally, this is treated as a supervised learning problem: you record thousands of videos, manually label every mistake, and train a classifier. This is incredibly expensive, time-consuming, and domain-specific—a "mistake" in a kitchen looks nothing like a "mistake" on a factory floor.
This paper tackles unsupervised mistake detection. It leverages a fundamental psychological insight: in skilled activities, our eyes lead our hands. Our gaze patterns are highly predictable when we know what we're doing. When we mess up, our attention often becomes erratic or deviates from the "norm." The authors use this deviation as a signal for error detection.
Methods
The core of their approach is a novel task called gaze completion.
As shown in Figure 1, the system takes a video and the initial part of a gaze trajectory. It then predicts how that gaze trajectory should continue if the action were being performed correctly.
The Gaze Completion Model
The authors built a transformer-based encoder-decoder architecture. They compared two ways to integrate gaze data with visual information:
- Channel Fusion: Adding gaze heatmaps as an extra input channel to the video frames.
- Correlation Fusion: A more sophisticated approach that explicitly models the relationship between "gaze tokens" and "visual tokens" (objects in the scene).
Detecting the "Misstep"
At test time, the model predicts the future gaze trajectory. The system then compares this prediction to the actual gaze recorded by the user's smart glasses. If the two trajectories are significantly different (measured via metrics like Euclidean distance or Heatmap likelihood), the system flags a "mistake."
Datasets
The researchers evaluated their method on three major egocentric datasets:
- EPIC-Tent: 7 hours of users assembling camping tents.
- HoloAssist: A large-scale dataset of users performing various tasks with expert assistance.
- IndustReal: Focused on procedural tasks in industrial-like environments.
Key Insights and Ideas
- Gaze as a Proxy for Intent: Because gaze is so tightly coupled with skilled action, it acts as a natural supervisory signal. You don't need labels because the "normal" behavior is inherently encoded in how people look at what they are doing.
- The "Novice" Advantage: The method is particularly effective for users with low self-rated confidence (novices). Their gaze patterns tend to be more erratic when they are unsure, making mistakes easier to spot compared to experts who might make "cleaner" errors.
- Complexity Matters: Gaze-based detection works best for actions requiring high hand-eye coordination (like "Object Manipulation") and less well for simple tasks like "Inspection," where gaze naturally wanders more.
Key Results
The proposed method achieved significant performance jumps over existing unsupervised baselines:
- Relative gains: +14% on EPIC-Tent, +11% on HoloAssist, and +5% on IndustReal.
- Supervised Parity: Remarkably, the unsupervised gaze-based method matched the performance of fully supervised models on some benchmarks, despite never seeing a "mistake" label during training.
- Competition Success: The method ranked first on the HoloAssist Mistake Detection challenge.
As seen in Figure 4, the "success" rate of the model is highest for Hand-Eye Coordination and Object Manipulation tasks, confirming that gaze is a highly reliable indicator for these specific types of skilled labor.
Limitations and Future Work
While powerful, the method has its limits. It struggles with "trivial" tasks where gaze isn't strictly necessary for execution. Furthermore, the current setup requires a gaze tracker at inference time to compare the user's actual gaze against the model's prediction.
Future work could involve integrating other signals—like hand motion or tool use—to create a more robust multi-modal mistake detector that doesn't rely solely on eye-tracking. Additionally, exploring how to provide the right kind of AR feedback once a mistake is detected remains an open human-computer interaction challenge.
Summary updated