Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities

Michele Mazzamuto, Antonino Furnari, Yoichi Sato, Giovanni Maria Farinella

TL;DR

Researchers have developed a way to detect mistakes in skilled activities (like assembling a tent or repairing machinery) using only eye-gaze data from egocentric videos. By training a model to "complete" a partial gaze trajectory, they can identify errors when a user's actual gaze deviates from the predicted "normal" pattern—all without needing a single manual mistake label.

What Problem Does This Paper Address?

Detecting when someone makes a mistake while performing a complex task is a holy grail for Augmented Reality (AR) assistants. If a wearable device knows you've missed a step or are confused, it can proactively offer help.

Traditionally, this is treated as a supervised learning problem: you record thousands of videos, manually label every mistake, and train a classifier. This is incredibly expensive, time-consuming, and domain-specific—a "mistake" in a kitchen looks nothing like a "mistake" on a factory floor.

This paper tackles unsupervised mistake detection. It leverages a fundamental psychological insight: in skilled activities, our eyes lead our hands. Our gaze patterns are highly predictable when we know what we're doing. When we mess up, our attention often becomes erratic or deviates from the "norm." The authors use this deviation as a signal for error detection.

Methods

The core of their approach is a novel task called gaze completion.

Figure 1. Top: gaze trajectories of a correct and wrong execution of the “place water container” action, together with gaze fixation maps averaged across many action instances. Note the higher vari- ability exhibited by wrong executions. Bottom: (a) The proposed unsupervised mistake detection method assumes as input a video with a partial gaze trajectory on the initial part of the video. (b) A gaze completion model predicts a gaze trajectory for the remain- ing part of the video, conditioned on the input video and the partial trajectory. (c) A mistake is detected if the predicted trajectory is significantly different from the observed one, suggesting a devia- tion from the expected attention patterns.

As shown in Figure 1, the system takes a video and the initial part of a gaze trajectory. It then predicts how that gaze trajectory should continue if the action were being performed correctly.

The Gaze Completion Model

The authors built a transformer-based encoder-decoder architecture. They compared two ways to integrate gaze data with visual information:

  1. Channel Fusion: Adding gaze heatmaps as an extra input channel to the video frames.
  2. Correlation Fusion: A more sophisticated approach that explicitly models the relationship between "gaze tokens" and "visual tokens" (objects in the scene).
Figure 2. The model takes as input F RGB frames Vt−F :t and a partial 2D gaze trajectory Tt−F :t−F/2 on the first F/2 input frames, and outputs a predicted trajectory ˆTt−F/2:t from the input video, conditioned on the input trajectory. The input trajectory is encoded as a spatio-temporal heatmap Q. Trajectory and RGB inputs are fused using two strategies, channel fusion, which adds gaze heatmaps as a separate channel (1), and correlation fusion, which uses a dedicated gaze-visual correlation module (2). We follow the design of [19] and process our inputs with a transformer encoder-decoder architecture which outputs a predicted gaze heatmap ˆQ, supervised via a standard Kullback–Leibler divergence loss using the ground truth unobserved trajectory Tt−F/2:t. The output trajectory ˆTt−F/2:t is recovered from ˆQ using a peak finding operation.

Detecting the "Misstep"

At test time, the model predicts the future gaze trajectory. The system then compares this prediction to the actual gaze recorded by the user's smart glasses. If the two trajectories are significantly different (measured via metrics like Euclidean distance or Heatmap likelihood), the system flags a "mistake."

Datasets

The researchers evaluated their method on three major egocentric datasets:

  • EPIC-Tent: 7 hours of users assembling camping tents.
  • HoloAssist: A large-scale dataset of users performing various tasks with expert assistance.
  • IndustReal: Focused on procedural tasks in industrial-like environments.

Key Insights and Ideas

  • Gaze as a Proxy for Intent: Because gaze is so tightly coupled with skilled action, it acts as a natural supervisory signal. You don't need labels because the "normal" behavior is inherently encoded in how people look at what they are doing.
  • The "Novice" Advantage: The method is particularly effective for users with low self-rated confidence (novices). Their gaze patterns tend to be more erratic when they are unsure, making mistakes easier to spot compared to experts who might make "cleaner" errors.
  • Complexity Matters: Gaze-based detection works best for actions requiring high hand-eye coordination (like "Object Manipulation") and less well for simple tasks like "Inspection," where gaze naturally wanders more.

Key Results

The proposed method achieved significant performance jumps over existing unsupervised baselines:

  • Relative gains: +14% on EPIC-Tent, +11% on HoloAssist, and +5% on IndustReal.
  • Supervised Parity: Remarkably, the unsupervised gaze-based method matched the performance of fully supervised models on some benchmarks, despite never seeing a "mistake" label during training.
  • Competition Success: The method ranked first on the HoloAssist Mistake Detection challenge.
Figure 4. Distributions of Correct/Wrong pred. by action type.

As seen in Figure 4, the "success" rate of the model is highest for Hand-Eye Coordination and Object Manipulation tasks, confirming that gaze is a highly reliable indicator for these specific types of skilled labor.

Limitations and Future Work

While powerful, the method has its limits. It struggles with "trivial" tasks where gaze isn't strictly necessary for execution. Furthermore, the current setup requires a gaze tracker at inference time to compare the user's actual gaze against the model's prediction.

Future work could involve integrating other signals—like hand motion or tool use—to create a more robust multi-modal mistake detector that doesn't rely solely on eye-tracking. Additionally, exploring how to provide the right kind of AR feedback once a mistake is detected remains an open human-computer interaction challenge.

Summary updated

Abstract

We address the challenge of unsupervised mistake detection in egocentric video of skilled human activities through the analysis of gaze signals. While traditional methods rely on manually labeled mistakes, our approach does not require mistake annotations, hence overcoming the need of domain-specific labeled data. Based on the observation that eye movements closely follow object manipulation activities, we assess to what extent eye-gaze signals can support mistake detection, proposing to identify deviations in attention patterns measured through a gaze tracker with respect to those estimated by a gaze prediction model. Since predicting gaze in video is characterized by high uncertainty, we propose a novel gaze completion task, where eye fixations are predicted from visual observations and partial gaze trajectories, and contribute a novel gaze completion approach which explicitly models correlations between gaze information and local visual tokens. Inconsistencies between predicted and observed gaze trajectories act as an indicator to identify mistakes. Experiments highlight the effectiveness of the proposed approach in different settings, with relative gains up to +14%, +11%, and +5% in EPIC-Tent, HoloAssist and IndustReal respectively, remarkably matching results of supervised approaches without seeing any labels. We further show that gaze-based analysis is particularly useful in the presence of skilled actions, low action execution confidence, and actions requiring hand-eye coordination and object manipulation skills. Our method is ranked first on the HoloAssist Mistake Detection challenge.
BibTeX
@inproceedings{mazzamuto2025gazing,
  author = {Michele Mazzamuto and Antonino Furnari and Yoichi Sato and Giovanni Maria Farinella},
  title = {{Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00778},
}
Figure