An assistant that watches a person assemble or set up equipment through a headset camera has to answer a simple question over and over: is this step actually done, and done correctly? Current vision-language models, which answer questions about images in text, handle this poorly. Handing them the full instruction graph barely helps. On the ProMQA-Assembly benchmark, a text-only model that never sees the video scores nearly as well as the best multimodal model. The models fall back on the wording of the instructions and routinely claim steps remain when the assembly is already finished. Yet when the visual state is identified correctly, language-model reasoning over procedures improves sharply. We propose to use a technical manual differently. Instead of pasting it into the prompt as background to reason over, we compile it ahead of time into short, visually checkable conditions for each step ("the red cable is seated in port B", "the battery's flat end faces the spring"). At run time, the assistant looks up the few conditions relevant to the current moment and asks the vision model to verify each one on a small image region chosen by the wearer's gaze and hands.
The headset imposes the constraints that shape this design. Standalone headsets overheat and drain their batteries under continuous vision-language inference. Their narrow field of view makes a "wall of text" unusable. The same compiled conditions address both problems. The system spends visual queries only at moments that matter, and when a check fails, what it displays is the failed condition itself: a single short phrase anchored in space, not a generated paragraph.
Core Claim
Our claim is that, for procedural assistance, the knowledge in documentation is most useful when turned into targeted perceptual checks, not supplied as context for reasoning. With the retrieved text held fixed, asking the vision model a handful of manual-derived yes/no questions about gaze-selected image crops should detect step completion and execution mistakes better than giving the same text to the model as context. It should also do better than letting the model write its own verification questions. It should manage this at a fraction of the inference calls and display text of continuous or dialog-based alternatives.
Each of the four directions plays a distinct role. The multimodal reasoning gap motivates the design: models see parts of a scene but fail to turn what they see into correct procedural conclusions. Domain knowledge supplies the content of the checks. Retrieval supplies the indexing and the decision of when to look something up. Hardware ergonomics sets the budget and the output format.
Where This Sits
Two findings motivate the work, and they look contradictory. ProMQA-Assembly builds 391 video question-answer pairs over Assembly101 recordings with instruction graphs given in the prompt. It finds text-only DeepSeek-R1 at 46.3 against multimodal Claude 3.7 Sonnet at 50.8 and humans at 67.5, with frequent failures to recognize that assembly is complete. PREGO pairs an online action recognizer with a language model that predicts the next step, and reports that oracle action labels dramatically improve the language model's reasoning. Read together, the two papers say the bottleneck is extracting the right visual evidence, not reasoning over the procedure. Our design targets exactly that extraction step.
Two results constrain the design. TheoremQA shows that adding theorem definitions to the prompt gives no significant gain, because models fail to apply rules, not to recall them. This argues against retrieval that merely appends manual passages. Plot Twist shows that vision-language models are weak at elementary visual reading and fragile to small changes such as added grid lines. That warns us that yes/no verification is not free. Checks must be narrow, applied to crops, and aggregated across frames rather than trusted one frame at a time.
The work closest to ours is Transparent and Coherent Procedural Mistake Detection. There, a vision-language model runs a self-dialog: it writes its own questions about a single frame and answers them, and preference tuning improves question relevance to as high as 95.0%. We differ in three ways. Our questions come from retrieved documentation rather than the model's own knowledge, which is where the domain knowledge gap bites. We evaluate over time rather than on single frames. And we operate under a fixed budget of queries rather than an open-ended dialog, which that paper itself identifies as too slow for wearable use.
Several papers supply methods we reuse. RaDIO retrieves only when a learned detector flags likely hallucination; it is text-only, and we adapt its trigger-when-uncertain logic to visual step tracking. Synthesize Step-by-Step breaks chart questions into atomic sub-questions paired with executable rationales, raising accuracy on ChartQA's human-written split from 38.2% to 53.9%. We apply the same kind of decomposition offline to manual steps. HoloAssist provides synchronized gaze and 3D hand pose from HoloLens 2 recordings, and shows that these signals raise intervention-prediction precision from about 13% to about 48%. We use them to choose where to crop.
On the ergonomic side, ARTiST finds that headset users prefer short, even fragmentary, instructions to long ones, and reports roughly 2-second API latency. the ventilator setup trial documents overheating and battery exhaustion on HoloLens 2 during clinical procedures. SARA uses gaze dwell time to decide when to call a language model during reading. These papers set our output format and our trigger design; we do not reproduce their user studies.
Research Program
The system has an offline stage and an online stage.
Offline, a language model reads each task's documentation: instruction graphs, step text, and where available the manufacturer manual with its diagrams. For every step it writes a small set of atomic conditions, each a short declarative phrase that could in principle be checked from one or a few frames. These split into preconditions (what must be true before the step) and completion conditions (what is true once it is done). A human spends a few minutes per task correcting the list, and we report how much correction was needed. The conditions are indexed by step and also by text and image embeddings, so they can be found when the current step is unknown.
Online, a lightweight step tracker runs continuously. We use existing recognizers, the online action detector from PREGO or a TGML-style graph tracker (Differentiable Task Graph Learning), and track which steps are plausibly in progress.
Retrieval fires at two kinds of moment: when the tracker's distribution over current steps becomes uncertain, and when the tracker claims a step has just finished. Formally, we query when the entropy $H(p_t)$ of the tracker's step distribution $p_t$ at time $t$ exceeds a threshold $\tau$, or when the most likely step changes. The threshold is set so that the average number of vision-model calls per minute stays under a budget $B$. $B$ is our proxy for the energy, heat, and latency cost that the ergonomics literature documents.
When a query fires, the system retrieves the conditions for the one to three most probable steps. It crops the frame around the gaze fixation and the hand-object contact region, and asks a frozen vision-language model each condition as a yes/no question on the crop. It combines answers across a short window of frames by majority vote.
The outputs follow directly. A step counts as complete only when all its completion conditions hold. A mistake is flagged when a precondition fails at the start of a step, or when a completion condition fails after the tracker declares the step done. The headset displays the failed condition verbatim, capped at a fixed word budget and anchored at the gaze location. There is no free-form explanation generation.
Three analyses are central. The first tests whether gains come from the content of the conditions or from their verification format, which the context-versus-probe ablation below isolates. The second checks whether cropping by gaze and hand pose matters beyond full-frame questions. The third traces how accuracy trades off against the budget $B$ as it varies from a few calls per minute up to continuous querying.
Baselines
The do-nothing reference is the step tracker alone, with no retrieval and no verification. It declares completion and mistakes from action recognition and order checks. Running it on the same recordings and with the same recognizer shows how much the verification layer adds. For ProMQA-Assembly, the matching naive references are the reported text-only score of 46.3 for DeepSeek-R1 and the human score of 67.5 (ProMQA-Assembly).
The strongest in-context competitor is the ProMQA setting itself. A frontier vision-language model receives the full instruction graph and the video and answers directly. Claude 3.7 Sonnet reports 50.8 there. We will rerun this setting with the same vision-language model used inside our system, so the comparison isolates how knowledge is used rather than which model is stronger. Beating it means a higher score with the same model, and specifically fewer errors on "is it done" questions.
The strongest mistake detectors differ by benchmark. On HoloAssist it is the Video Q-Former system of Patsch et al., a supervised model that detects mistakes and calls a language model only when an alert fires. It reports an F1 of 55.0, against GazeCompl at 51.0 and an RGB-only TSformer at 35.1. On Assembly101-O it is TGML (Differentiable Task Graph Learning). TGML reports a +19.8% gain over PREGO, which itself reports an F1 of 41.8 (PREGO). The file does not state TGML's absolute F1, so we will reproduce it. We run TGML with the identical recognizer our system uses. Beating it means higher F1 than TGML alone when our verification is layered on top.
Two ablations isolate the key ingredient. In the context ablation, the retrieved conditions are pasted into the vision-language model's prompt as text, with the full frame, and the model answers the task question directly. Content is identical to our method; only the format differs. In the self-questioning ablation, modeled on Storks et al., the model writes its own verification questions, capped at the same number of questions as our method. All systems receive the same budget $B$ of vision-model calls, the same frames, and the same threshold-tuning budget on validation splits.
Why not just…?
Why not just put the whole manual and video into a long-context model? ProMQA-Assembly is close to that experiment already: the graph was in context, and the best model reached 50.8 against a human 67.5 while hallucinating remaining steps. The failure is in reading the scene, which more context does not fix. The context ablation above tests this directly, since it gives the model exactly the retrieved text without the probing format. If that ablation matches our full method, our claim is wrong and we will report it as such.
Why not just use a learned task graph like TGML? On benchmarks whose mistakes are mostly about step order, this is a real threat. TGML may match us on Assembly101-O, because order violations are what its graph encodes. Its authors acknowledge it cannot see how a step was executed, such as a part inserted backwards or a step left half-finished. We therefore report results separately for order mistakes and execution-state mistakes. We claim gains only where the graph is blind, and we run our verification on top of TGML rather than instead of it.
Why not just let the model generate its own verification questions? Storks et al. show that self-generated questions can become highly relevant after preference tuning. But the model can only ask about what its parametric knowledge suggests, and TheoremQA and the domain-gap literature indicate this is exactly what fails in specialized procedures. The budget-matched self-questioning ablation settles the question. If self-generated questions match manual-derived ones on HoloAssist device tasks, where manuals contain specific connector and orientation details, then retrieval adds nothing and the contribution reduces to cropping and budgeting.
Why not just fine-tune a supervised mistake detector? The Patsch et al. detector is strong in-distribution. But it needs labeled mistakes for each task, and mistakes in physical work are open-ended, the point PREGO makes. We test transfer by holding out HoloAssist tasks: the supervised detector is trained on the remaining tasks, while our method only receives the held-out tasks' manuals.
Proposed Experiments
The first experiment uses ProMQA-Assembly. We compile conditions from its 78 instruction graphs and report overall score plus accuracy on the subset of completion questions. The comparison is against our rerun of the in-context setting with the same vision-language model.
The second covers online mistake detection. On Assembly101-O and Epic-tent-O we compare the tracker alone, TGML, and TGML plus verification, with results broken down by mistake type. On HoloAssist, for which we will gather manufacturer manuals for the 20 tasks, we compare against the Patsch et al. detector both in-distribution and on held-out tasks.
Across all benchmarks, we plot accuracy against the per-minute call budget $B$ and report word counts of displayed output. Ego4D-PMD is used for the self-questioning comparison, since it is the benchmark on which that method was developed; its reported human accuracy is about 72.0%, and the file gives no scalar result for GPT-4o on it.
The decisive check is the context-versus-probe ablation on completion questions and execution mistakes. If probing with manual-derived conditions does not beat pasting the same conditions into the prompt, at equal model and equal budget, the central claim fails.
A secondary check addresses the fragility documented in Plot Twist. We measure per-condition verification accuracy against human labels on a sample of crops, which shows whether errors come from bad conditions or bad perception.
Finally, a small pilot on HoloLens 2 compares a 2 to 3 word failed-condition display against generated explanations on task time and NASA-TLX workload. We frame it only as a sanity check on the output format, not a full user study.
References
- ProMQA-Assembly: Multimodal Procedural QA Dataset on Assembly
- PREGO: online mistake detection in PRocedural EGOcentric videos
- TheoremQA: A Theorem-driven Question Answering Dataset
- Plot Twist: Multimodal Models Don't Comprehend Simple Chart Details
- Transparent and Coherent Procedural Mistake Detection
- RaDIO: Real-Time Hallucination Detection with Contextual Index Optimized Query Formulation for Dynamic Retrieval Augmented Generation
- Synthesize Step-by-Step: Tools, Templates and LLMs as Data Generators for Reasoning-Based Chart VQA
- HoloAssist: an Egocentric Human Interaction Dataset for Interactive AI Assistants in the Real World
- ARTiST: Automated Text Simplification for Task Guidance in Augmented Reality
- An Augmented Reality–Based Guide for Mechanical Ventilator Setup: Prospective Randomized Pilot Trial
- SARA: Smart AI Reading Assistant for Reading Comprehension
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos
- Technical Report for Egocentric Mistake Detection for the HoloAssist Challenge