When a vision system watches someone cook or assemble furniture, it usually works in two stages. First a detector turns pixels into symbols such as "egg in pot" or "step 3 done". Then a reasoner checks whether the sequence of symbols makes sense. If the detector misses the moment the egg went into the pot, every later step that depends on it looks impossible. The reasoner then reports a cascade of "mistakes" that the person never made. The reasoner already notices that something is off: in the language of infant-cognition research, it is surprised, meaning it has seen an event its model of the task says should not happen. Yet no current pipeline uses that surprise to send the detector back to check. This note proposes to do exactly that, and to treat the hard part honestly. A surprise can mean the person really made a mistake, or it can mean the system misperceived. Telling these apart is the whole problem.
The core idea fits in two sentences. When a long-horizon model of the task signals a violated expectation, we re-run perception in a targeted way: we prompt the detector for the specific objects and the specific time window that the violated precondition names. If the new evidence makes the surprise go away, we blame perception and repair the state. If the surprise persists, we report a real mistake. The second ingredient is elapsed time. The expected state is conditioned on continuous elapsed time (Δt) rather than on step counts, because "boil for eight minutes" and "idle for eight minutes" differ only in what happened during the gap. Elapsed time also tells the system how far back to look.
Core Claim
Downstream surprise, scored by a task-state model that sees elapsed time, is a better trigger for spending extra perception compute than the detector's own confidence or a uniform increase in detection density. Surprise points to the frames where an error will cascade, not merely where the detector is unsure. The measurable consequence is fewer cascaded false alarms in online procedural mistake detection while true mistakes are kept. We predict the gain will be concentrated on omission errors, which no amount of downstream reasoning can recover without new pixel evidence.
Where This Sits
Four strands of prior work bear on the proposal, each in a different role.
The violation-of-expectation literature supplies the scoring rule and the "explain-away" logic. Smith et al. define surprise as the negative log-likelihood of an observation under a physics model's forecast. X-VoE goes further: when prediction error spikes, its XPL learner hypothesizes hidden objects that would reconcile the anomaly. XPL is the closest conceptual relative. It differs from this proposal in three ways. It explains anomalies by imagining latent content inside its own model rather than by re-querying a real detector. It works on synthetic rolling-ball scenes. And it has no notion of an anomaly that should not be explained away, whereas in mistake detection some anomalies are the signal itself.
Procedural mistake detection supplies the setting and the strongest baselines, and it motivates the gap. Differentiable Task Graph Learning learns a precondition graph over steps and flags online violations. Its own limitations note that misrecognized actions flow straight into graph errors, and that it cannot express duration-dependent preconditions. Action Effect Modeling shows that post-action object relations, distilled from Grounding DINO and GPT-4o scene graphs during training, beat motion features. It judges each segment in isolation, so it cannot see a step-2 flaw invalidating step 15. The procedural assistant survey documents a "correctness bias" in general video-language models and reports that they fall below 60% on CaptainCook4D while specialized models reach about 72%. This constrains us: we should not make a general video-language model the reasoner.
Cascading-error work supplies both the diagnosis and the existing mitigations, and every one of those mitigations is feedforward. HAST-Graph2Seq prunes noisy detector edges with a learned similarity graph. R4DSG stops tracking errors from spreading by anchoring objects to furniture and cutting the timeline into episodes. Both reduce the damage of bad detections. Neither can recover a detection that was never made, which is the case Newell et al. identify as fundamental.
Temporal encoding supplies the method for elapsed time. Giuliari et al. show that sinusoidal encodings of continuous timestamps stay accurate with 75% of frames missing. TiPNN parameterizes periodic functions by the gap Δt between events. TSQA adds a time-order loss so that embeddings respect "before" and "after". We reuse the Δt parameterization and the order loss inside the task-state model. This is a supporting role, not the contribution.
Research Program
The system has three parts.
The first is a belief model over task state. It is a small transformer over the stream of perceived step and object-relation events, with each event's position encoded by sinusoidal features of elapsed time since every earlier event, following TiPNN. It is trained on correct demonstrations only, with a precondition structure initialized from a learned task graph as in DTGL. A TSQA-style order loss enforces chronological direction. At each new perceived event $o_t$ it outputs a surprise score
$$S_t = -\log p_\theta(o_t \mid o_{<t}, \Delta t_{<t}),$$
where $p_\theta$ is the belief model's predictive distribution, $o_{<t}$ the perceived history, and $\Delta t_{<t}$ the elapsed times between events.
The second part is targeted re-perception. When $S_t$ exceeds a threshold, the belief model's attention over the history (or, more directly, the unmet preconditions of $o_t$) names candidate missing or mislabeled events, each with an object vocabulary and a time window derived from Δt. We re-run an open-vocabulary detector (Grounding DINO) on that window, using a lower confidence threshold, denser frame sampling, and prompts restricted to the named objects. This produces revised evidence $o'$.
The third part is the attribution step. We compute the surprise of the revised history,
$$S'_t = -\log p_\theta(o_t \mid o'_{<t}, \Delta t_{<t}),$$
and also check that the revised evidence $o'$ is itself not improbable. Without that check, a lowered detector threshold could hallucinate a convenient egg. If $S'_t$ drops below threshold and the repair is plausible, the system updates its state and does not raise an alarm. Otherwise it reports a mistake. The plausibility check is the guard against explaining away real errors. We will study the trade-off it creates explicitly, by sweeping thresholds and plotting true-mistake recall against cascaded-false-alarm rate.
Our analysis tool is a perception-error ledger. EgoPER, CaptainCook4D, Assembly101-O and EPIC-Tent-O carry ground-truth step annotations. That lets us mark every flagged mistake as either a true annotated mistake, a direct perception error, or a cascaded false alarm, meaning an alarm whose cause is an earlier perception error. The cascaded-false-alarm rate is the quantity the method targets. Existing papers do not report it.
Baselines
The naive reference is the same pipeline with no re-perception: detector, then belief model, with alarms raised whenever surprise crosses threshold. This is our own system with the feedback loop cut, so it isolates the loop's contribution under identical training. A second reference is an oracle that feeds ground-truth step labels to the belief model. It bounds how much of the remaining error comes from perception at all. If the oracle gap is small on a dataset, no re-perception method can help there, and we will say so.
The strongest published method on EgoPER and CaptainCook4D is Action Effect Modeling. It reports 73.8% AUC and 66.7% error detection accuracy on EgoPER, and 62.5% AUC and 71.9% error detection accuracy on CaptainCook4D. We will run their released setup where available, or reimplement it with the same detector backbone, and report it under their protocol. Beating it means a higher AUC on both datasets under that protocol with the same Grounding DINO backbone. Separately from AUC, beating it also means a lower cascaded-false-alarm rate, which we compute for both systems from the ledger.
On Assembly101-O and EPIC-Tent-O the strongest method is DTGL's Direct Optimization model. The file gives only relative gains of +19.8% and +7.5% over the PREGO baseline, with no absolute values, so we will rerun DTGL with the same upstream action recognizer to get comparable numbers.
Two ablations isolate the key ingredients. The first replaces elapsed-time encoding with discrete step indices, which tests whether Δt matters for which surprises fire and which window gets re-examined. The second keeps re-perception but turns off the plausibility check, which measures how many real mistakes the loop explains away. All variants share the detector, the training data and the hyperparameter search budget. Every method that spends extra perception compute is held to the same count of extra detector calls per video.
Why not just…?
Why not just lower the detector threshold everywhere, or run it on every frame? This is a real threat, because it attacks omissions directly. Our answer has to be empirical. We give a uniform-density variant exactly the extra detector-call budget that the surprise loop uses, spread across the video, and compare cascaded-false-alarm rates. The case for surprise targeting is that uniform loosening adds false detections everywhere, and those also corrupt the downstream graph, as Iokawa et al. note for attribute errors. If uniform loosening matches us at equal budget, then the claim that surprise is a good trigger fails, and the paper becomes a negative result about targeting.
Why not just trigger re-examination from the detector's own uncertainty? Detector confidence measures how hard the pixels are. It does not measure whether an error will matter later. A confidently missed object, for example one fully occluded during the key moment, never triggers anything. Conversely, many uncertain detections are irrelevant to the task. We test this with a confidence-triggered variant at matched budget. We expect it to help with misclassifications but not with confident omissions, and we will break down the results by error type in the ledger to check.
Why not just decode the step sequence jointly under the task-graph prior, as in an HMM, and skip re-perception? This is the most serious alternative. If the action recognizer outputs full class probabilities, joint decoding can already correct many misclassifications without new evidence, and it is cheap. We include it as a baseline with the same belief model serving as the prior. The proposal's claim survives only where decoding cannot reach: steps the recognizer never registered, and object relations below the detector's threshold. Joint decoding can only infer such steps, and inferring a step that was not observed is exactly what erases real omission mistakes. Re-perception instead checks the pixels. If joint decoding equals the loop on the cascade metric, we will conclude that new evidence is unnecessary at current detector quality.
Why not just use a long-context video-language model? The survey reports that general models fall below 60% on CaptainCook4D and are biased toward judging procedures as correct (z1oong et al.). GRASP finds near-chance physical reasoning. We will report one such model zero-shot as a reference but do not expect it to compete.
Decisive Checks
The first check is a controlled cascade test. On correct-execution videos from EgoPER and CaptainCook4D, we delete a single ground-truth detection at varying depths in the procedure, mimicking an upstream miss. We then count the false alarms that follow. The prediction is that the feedforward baseline produces alarms that grow with the number of dependent later steps, while the surprise loop recovers most deletions, because the deleted object is still visible in the pixels. Recovery will be lower when the deletion falls in an occluded window, and we will report that case separately.
The second check is the matched-budget comparison of the three re-perception triggers: surprise, detector confidence and uniform density, plus joint decoding, all on natural data. The third is the explain-away audit: true-mistake recall with and without the plausibility check. Losing more than a few points of recall would show that the loop is suppressing the very errors it should report.
The elapsed-time ablation is judged on duration-dependent steps, such as boiling, resting and drying in the recipe datasets. There the hypothesis is specific: step-index encoding should miss "took it out too early" mistakes that the Δt-conditioned model catches.
References
- Modeling Expectation Violation in Intuitive Physics with Coarse Probabilistic Object Representations
- X-VoE: Measuring eXplanatory Violation of Expectation in Physical Events
- Differentiable Task Graph Learning: Procedural Activity Representation and Online Mistake Detection from Egocentric Videos
- Procedural Mistake Detection via Action Effect Modeling
- Building Egocentric Procedural AI Assistant: Methods, Benchmarks, and Challenges
- Hierarchical Attention Based Spatial-Temporal Graph-to-Sequence Learning for Grounded Video Description
- R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video
- Pixels to Graphs by Associative Embedding
- Transformer Networks for Trajectory Forecasting
- Temporal inductive path neural network for temporal knowledge graph reasoning
- Improving Time Sensitivity for Question Answering over Temporal Knowledge Graphs
- Visual-Textual Entailment with Quantities Using Model Checking and Knowledge Injection
- GRASP: A Novel Benchmark for Evaluating Language GRounding and Situated Physics Understanding in Multimodal Language Models