Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities

Michele Mazzamuto, Antonino Furnari, Giovanni Maria Farinella, Farinella, Giovanni Maria

Abstract

We address the challenge of unsupervised mistake detection in egocentric video of skilled human activities through the analysis of gaze signals. While traditional methods rely on manually labeled mistakes, our approach does not require mistake annotations, hence overcoming the need of domain-specific labeled data. Based on the observation that eye movements closely follow object manipulation activities, we assess to what extent eye-gaze signals can support mistake detection, proposing to identify deviations in attention patterns measured through a gaze tracker with respect to those estimated by a gaze prediction model. Since predicting gaze in video is characterized by high uncertainty, we propose a novel gaze completion task, where eye fixations are predicted from visual observations and partial gaze trajectories, and contribute a novel gaze completion approach which explicitly models correlations between gaze information and local visual tokens. Inconsistencies between predicted and observed gaze trajectories act as an indicator to identify mistakes. Experiments highlight the effectiveness of the proposed approach in different settings, with relative gains up to +14%, +11%, and +5% in EPIC-Tent, HoloAssist and IndustReal respectively, remarkably matching results of supervised approaches without seeing any labels. We further show that gaze-based analysis is particularly useful in the presence of skilled actions, low action execution confidence, and actions requiring hand-eye coordination and object manipulation skills. Our method is ranked first on the HoloAssist Mistake Detection challenge.

Introduction

As wearable technology like smart glasses and Augmented Reality (AR) headsets becomes more integrated into professional and domestic environments, the demand for intelligent assistants capable of providing real-time feedback grows. A critical component of such assistance is the ability to detect when a user makes a mistake during a complex task. Traditional mistake detection methods rely heavily on supervised learning, requiring extensive manual labeling of "correct" versus "erroneous" actions. This labeling process is not only labor-intensive but also domain-specific, limiting the generalizability of models across different tasks.

This work proposes an unsupervised approach to mistake detection by leveraging eye-gaze signals. The central hypothesis is that in skilled human activities, eye movements are highly functional and predictable, tightly coupled with hand-eye coordination. When a mistake occurs, this coordination is disrupted, leading to gaze patterns that deviate from expected "correct" behavior.

Overview of the gaze-based mistake detection framework, illustrating how deviations from predicted gaze trajectories signal mistakes.

Methodology: Gaze Completion

The core technical contribution is the Gaze Completion task. Unlike standard gaze prediction, which estimates gaze from visual frames alone and often suffers from high uncertainty, gaze completion predicts the future portion of a gaze trajectory given the initial segment and the corresponding video sequence.

Model Architecture

The proposed architecture utilizes a transformer-based encoder-decoder framework. Key components include:

  1. Token Embedding: Visual frames and partial gaze trajectories are converted into tokens.
  2. Gaze-Visual Correlation: A novel module that explicitly models the relationship between local visual features (visual tokens) and gaze information (gaze tokens).
  3. Transformer Decoder: Generates a predicted gaze heatmap sequence ($\hat{Q}$).
Detailed architecture of the Gaze Completion model, featuring the Gaze-Visual Correlation module and transformer-based processing.

Mistake Detection Logic

At inference time, the model predicts the "ideal" gaze trajectory for a skilled execution of the task. This prediction is compared against the actual gaze data captured by the wearable device. Inconsistencies—measured as the negative log-likelihood of the observed gaze on the predicted heatmap—serve as an indicator of a mistake.

Experimental Results

The method was evaluated on three diverse egocentric datasets: EPIC-Tent (tent assembly), HoloAssist (AR-guided physical tasks), and IndustReal (industrial assembly).

Performance and Benchmarking

The unsupervised gaze-completion approach demonstrated significant improvements over existing unsupervised baselines, with relative gains of +14% on EPIC-Tent, +11% on HoloAssist, and +5% on IndustReal. Notably, the model achieved first place in the HoloAssist Mistake Detection challenge and performed competitively with fully supervised state-of-the-art models without requiring any mistake labels during training.

Action Specificity and Insights

Analysis revealed that gaze-based detection is most effective for actions requiring high hand-eye coordination and object manipulation. In these scenarios, gaze is a sensitive proxy for cognitive state and procedural flow. Conversely, the method is less effective for simple inspection tasks where gaze patterns are naturally more variable and less coupled to specific motor actions.

Conclusion

By framing mistake detection as a deviation from predicted functional gaze, this research provides a scalable, unsupervised solution for intelligent egocentric assistants. The results underscore the value of eye-gaze as a rich source of information for understanding human skill and identifying procedural errors in real-world tasks. Future work may explore integrating gaze with other modalities, such as 3D object tracking, to further enhance detection robustness.

Summary updated

Abstract

We address the challenge of unsupervised mistake detection in egocentric video of skilled human activities through the analysis of gaze signals. While traditional methods rely on manually labeled mistakes, our approach does not require mistake annotations, hence overcoming the need of domain-specific labeled data. Based on the observation that eye movements closely follow object manipulation activities, we assess to what extent eye-gaze signals can support mistake detection, proposing to identify deviations in attention patterns measured through a gaze tracker with respect to those estimated by a gaze prediction model. Since predicting gaze in video is characterized by high uncertainty, we propose a novel gaze completion task, where eye fixations are predicted from visual observations and partial gaze trajectories, and contribute a novel gaze completion approach which explicitly models correlations between gaze information and local visual tokens. Inconsistencies between predicted and observed gaze trajectories act as an indicator to identify mistakes. Experiments highlight the effectiveness of the proposed approach in different settings, with relative gains up to +14%, +11%, and +5% in EPIC-Tent, HoloAssist and IndustReal respectively, remarkably matching results of supervised approaches without seeing any labels. We further show that gaze-based analysis is particularly useful in the presence of skilled actions, low action execution confidence, and actions requiring hand-eye coordination and object manipulation skills. Our method is ranked first on the HoloAssist Mistake Detection challenge.
BibTeX
@inproceedings{mazzamuto2025gazing,
  author = {Michele Mazzamuto and Antonino Furnari and Giovanni Maria Farinella and Farinella, Giovanni Maria},
  title = {{Gazing Into Missteps: Leveraging Eye-Gaze for Unsupervised Mistake Detection in Egocentric Videos of Skilled Human Activities}},
  booktitle = {IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  year = {2025},
  doi = {10.1109/cvpr52734.2025.00778},
}
Figure