Online Mistake Detection in Instruction-Conditioned Procedural Video

Amit Feldman, Ilan Shimshoni, Rotem DrorarXiv 2026 · arXiv

TL;DR

Researchers have developed a new system for detecting mistakes in procedural videos (like cooking or assembly) in real-time by comparing actions against specific written instructions. By using a joint Vision-Language Model (VLM) approach, the system can identify not just that a mistake happened, but exactly which step went wrong and why, significantly outperforming previous methods that lacked instruction-awareness.

What Problem Does This Paper Address?

Imagine a smart assistant watching you cook a complex recipe. To be truly helpful, it needs to tell you that you've missed a step or used the wrong ingredient while you're doing it, not after the meal is ruined.

Current AI for "mistake detection" faces two major hurdles:

  1. Offline vs. Online: Most high-performing models are "offline," meaning they need to see the entire video before making a judgment. This is useless for real-time intervention.
  2. Instruction Blindness: Many "online" models detect deviations from a general "normal" sequence but don't actually read the specific instructions for the task at hand. Visual evidence alone is often ambiguous—pouring water into a bowl is a perfectly "normal" action, but it's a mistake if the instructions specifically said to pour it into a bowl already containing oats.

This paper bridges the gap by proposing an online, instruction-conditioned system that monitors egocentric (first-person) video as it unfolds.

Methods

The authors compare two architectural philosophies for this task:

  • Modular Pipeline: A three-stage chain where one module identifies the attempted step, the next localizes it in time (start/end), and the third verifies if the execution was correct.
  • Joint Model: A single, fine-tuned Vision-Language Model (based on Qwen3.8-27B) that predicts the step identity, temporal boundaries, execution class, and a natural language rationale all in one response.
Figure 1. Overview of the two approaches. At each call, the system receives the current window V [τ, t] (sampled at 1 fps), the task description d, the instruction list I, and its prediction history. (a) The modular approach chains a Step Identifier, which selects an instruction s ∈I or EXTRA; a Temporal Localizer, which predicts the attempt’s boundaries [a, b]; and an Execution Verifier, which classifies the cropped attempt V [τ+a, τ+b] as CORRECT, MODIFICATION, SLIP, or CORRECTION, using its previous execution labels as context. Extra actions are labeled ADDITION without verification. (b) The joint approach predicts all fields in a single structured response, conditioned on up to 12 previous predictions. In both, a NOT COMPLETED output (dashed) keeps the window start τ and extends the window by ∆= 5 s; an accepted record updates the history and starts the next window at the predicted end.

Both approaches use an incremental windowing strategy. The system looks at a 5-second snippet; if it doesn't see a completed action, it extends the window by another 5 seconds. Once an action is "accepted," the history is updated, and the process starts over from the end of that action.

The models classify actions into a fine-grained taxonomy: Correct, Modification (different technique), Slip (failed goal), Correction (fixing a previous slip), or Addition (extra step). Omissions are detected at the very end of the video.

Datasets

The study utilizes EgoPER, a dataset of 389 egocentric videos covering five cooking tasks (e.g., making oatmeal, tea, or quesadillas).

Crucially, the authors found that the original EgoPER split only put erroneous videos in the test set, making supervised learning of mistakes impossible. They created a new task- and error-stratified split that redistributes erroneous videos into training and validation sets, allowing the models to learn what "slips" and "modifications" actually look like.

Key Insights and Ideas

  • The "Joint" Advantage: The joint model significantly outperformed the modular pipeline. The reason? Error propagation. In a modular setup, if the first module misidentifies the step, the localizer and verifier are essentially guessing. The joint model allows these decisions to inform each other simultaneously.
  • Slips are the Hardest: "Slips" (where an action is attempted but fails to achieve the goal, like pouring water into the wrong bowl) saw the biggest improvement with the joint approach (56.4% vs 37.5% F1). This suggests that understanding the intent behind an action requires the holistic view provided by a joint model.
  • Instruction Matters: By conditioning on text, the model can distinguish between a "Modification" (using a knife instead of a spoon to stir—often fine) and a "Slip" (missing the target bowl—requires correction).

Key Results

The joint model emerged as the clear winner, particularly in detecting actual errors rather than just segmenting the video.

  • Binary Mistake Detection: The joint model achieved an F1 score of 84.1%, compared to 80.7% for the modular approach.
  • Omission Detection: The joint model was highly effective at identifying skipped steps, reaching a 95.4% Omission-IoU.
  • Comparison to Baselines: On a subset of videos containing mistakes, both proposed designs significantly outperformed existing state-of-the-art models like AEM and ESTANet, which often struggled to localize errors or lacked the context to identify fine-grained mistake types.
Table 2. Error detection (%). EDA denotes error detection accu- racy; [email protected] is the macro-average of error and normal F1, with segments matched at temporal IoU 0.5. O-Acc is Omission de- tection accuracy and O-IoU is omission IoU. Zero-shot uses the untrained backbone with the joint approach’s prompt. AEM and ESTANet are evaluated only on the erroneous subset because the remaining test videos overlap with their training or validation data.

Qualitatively, the models provide "reasoning" for their decisions, such as "Dropped tortilla on floor," which is vital for a user-facing assistant.

Figure 2. Mistake predictions on two test videos. Our approaches assign a class to each segment; AEM and ESTANet give a binary error decision (pink). The number in each row is the binary [email protected] on that video. grey marks that no action is detected.

Limitations and Future Work

  • Domain Specificity: The models were only tested on five cooking tasks. It remains to be seen how well they generalize to vastly different procedures like furniture assembly or medical protocols.
  • Real-Time Latency: While the logic is "online," the paper uses a replay-based evaluation. True real-time deployment on edge devices would require further optimization of the large VLM backbone.
  • Human-in-the-loop: The current study doesn't measure how a human would actually react to these alerts or if the alerts are delivered quickly enough to prevent a "point of no return" in the procedure.

Summary updated

Abstract

An assistant monitoring a person performing a procedure is most useful when it can detect mistakes while there is still time to recover. This requires judging actions against task instructions as the video unfolds, identifying the attempted step, locating it in time, and assessing its execution. Existing online detectors do not use the instruction text, while instruction-conditioned methods operate offline. In this paper, we study online, instruction-conditioned mistake detection in egocentric videos. Using only observed video, task instructions, and prior predictions, our model identifies and localizes completed action attempts and classifies them as correct, modified, slip, corrections, or additions; omitted steps are determined after the video ends. We compare a modular pipeline, with separate step identification, temporal localization, and execution verification, to a joint model that predicts all outputs in a single step. To support supervised learning from mistakes, we repartition EgoPER to include erroneous videos in all splits. The approaches achieve similar step-segmentation [email protected] (93.6% and 92.1%), while the joint model achieves higher binary mistake-detection [email protected] (84.1% vs. 80.7%), with its largest gain on slips (56.4% vs. 37.5%). Component-level analysis suggests that error propagation between stages contributes to the modular approach's lower detection score. On test videos containing mistakes, both designs outperform released AEM and ESTANet checkpoints under our evaluation protocol.
BibTeX
@misc{feldman2026online,
  author = {Amit Feldman and Ilan Shimshoni and Rotem Dror},
  title = {{Online Mistake Detection in Instruction-Conditioned Procedural Video}},
  year = {2026},
  eprint = {2611.25567},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.25567},
}
Figure