When an AI agent carries out a long task on screen, such as navigating several websites to book or compare something, the usual way to find where it went wrong is to hand the whole recorded run to a large model acting as a judge. This proposal starts from a suspicion the literature makes plausible but has not tested. The judge probably fails in the same way the agent does. Vision-language models pay less and less attention to the image as the text they are conditioning on grows. They also tend to treat earlier text in the conversation as true. A judge reading step 12 of a 13-step screenshot trajectory has already read eleven steps of the agent's own confident narration. It may simply believe the agent ("I clicked the checkout button and the cart page loaded") instead of checking the screenshot. If that is right, judge errors should rise with step position, and they should concentrate on the steps that matter most: the early, silent perception mistakes that later steps build on.
We propose to measure this "inherited hallucination" effect directly and to remove it with one design choice plus one piece of specialization. The design choice is to give the judge only a local view of each step: the screenshot before the action, the action, the screenshot after, and the agent's one-sentence claim about what changed, with no trajectory history. The judge then decides whether that claim is visually supported. A separate, purely symbolic pass links these local verdicts into a graph of which steps depend on which, and locates the earliest failure that later failures follow from. The specialization is a compact 7-billion-parameter vision-language judge trained on synthetic counterfactual transitions, meaning real before/after screenshot pairs paired with deliberately falsified claims. At that size it is cheap enough to run at every step and to trigger retries during execution.
Core Claim
We make two claims, one empirical and one about method. The empirical claim is that judges which see the full trajectory, including frontier models, accept false agent claims at a rate that increases with step index and with the number of prior agent assertions in context. This is the judge-side analogue of the context trap reported in Why LVLMs Are More Prone to Hallucinations in Longer Responses, where attention shifts from image tokens to generated text as length grows. It is also the analogue of the turn-20 visual-attention collapse in Visual Multi-Agent System, which showed that later agents treat earlier dialogue as ground truth.
The method claim is that local transition verification makes judge accuracy flat in step index. A small judge specialized for this one check can then match or beat a frontier full-context judge on step-level failure detection and root-cause localization, at a small fraction of the cost. Either claim could fail independently. Section "Why not just…?" treats the case where context windowing alone, without any training, does all the work.
Positioning Against Prior Work
The closest work falls into three groups. The first group established the value of step-level judging for long tasks. AgentEval models text-only agent runs as a graph and scores each node with a calibrated GPT-4o. It finds that 63% of step failures originate upstream and that step-level judging raises failure-detection recall from 0.41 to 0.89. BaT uses stage-level rubric judges as dense rewards to train 4B and 9B medical research agents. Both motivate this proposal, and both are text-only. AgentEval also judges each node with the upstream context visible, which is exactly the condition we hypothesize is harmful once screenshots enter the picture.
The second group built multimodal judges, all for single static image–response pairs. MLLM-as-a-Judge benchmarks multimodal models as judges. It reports that step-by-step reasoning can make them hallucinate visual details and then rationalize wrong verdicts, and that judges sometimes do better from text descriptions than from pixels. MR. Judge trains a Qwen2.5-VL 7B judge on reverse-synthesized negative responses containing injected hallucinations, and reaches 83.2% on VL-RewardBench's hallucination split. Flex-Judge shows that about 1,000 text-only reasoning samples transfer judging ability across modalities. We reuse MR. Judge's reverse-synthesis idea and Flex-Judge's cheap reasoning seed. The difference is the object being judged: a pair of screenshots plus a claimed state change, which none of these judges were built to check.
The third group studies long multimodal trajectories without step-level judges. MMInA documents a "long-chain penalty", where agents fail more on the first hop of a longer task, and names wandering as the dominant failure. It scores only task and hop success. Mind2Web 2 builds a frontier agent-as-judge with tree rubrics and leaf checks against cached screenshots, at 99.03% agreement with experts. Those leaf checks verify final answers against rubric items, not per-step state transitions, and they depend on frontier models. Adaptive Safety Probing is the closest in spirit: it detects when a robot policy acts without looking at its target and triggers a re-run, which resolves 70–80% of detected failures. We borrow that "detect, then retry" loop but apply it to a semantic check on web trajectories rather than to single short motor primitives.
Research Program
The first component is a measurement study of inherited hallucination. We take agent trajectories on MMInA, where steps average about 13 actions across 2.85 hops, and collect runs from several agents so that failures are diverse. Annotators label each step as supported or unsupported: does the after-screenshot actually show what the agent claims? They also mark root-cause steps. We then present each step to judges under matched conditions. In the full-history condition the judge sees all prior screenshots and agent text. In the local condition it sees only the before/after pair and the claim. In a history-without-claims condition, prior agent narration is removed but prior screenshots are kept.
The key quantity is the false-acceptance rate, defined as the probability the judge accepts a claim that annotators marked unsupported. We examine it as a function of step index and of the number of prior agent assertions consistent with the false claim. Comparing the history-without-claims condition to full history isolates whether the harm comes from the length of the visual context or from the agent's narration specifically. We will also log attention mass on image tokens in middle layers, following the measurement protocol of Visual Multi-Agent System, to check whether the behavioral effect has the attention signature they reported.
The second component is the specialized local judge. We build counterfactual training transitions from real before/after pairs on MMInA websites held out from evaluation, plus Mind2Web 2 cached screenshots. Each true claim gets falsified variants. Some name the wrong element. Some assert a page change that did not happen, which is the typical "wandering" signature. Some hallucinate an item or price. Some claim a goal was reached when it was not. A Qwen2.5-VL 7B backbone is first seeded with a Flex-Judge-style set of about 1,000 text reasoning samples. It is then trained with supervised fine-tuning and group-relative policy optimization, a reinforcement learning method that scores several sampled answers against each other, with a hard cap on reasoning length. We cap reasoning because the step-by-step-reasoning failure in MLLM-as-a-Judge and the length-dependent trap suggest that long reasoning inside the judge re-creates the problem we are removing. We treat reasoning length (none, capped, uncapped) as an experimental variable, not a fixed choice.
The third component is symbolic propagation and online use. Local verdicts, together with the dependency between each action and the page state it acted on, form a graph. The root cause of a failed run is the earliest unsupported transition that has a failed descendant; no model call is needed for this step. Online, the judge runs after each action. When it rejects a transition, the agent re-samples that action, following the retry policy of Adaptive Safety Probing. We measure whether this stops wandering and how much it lifts MMInA task success.
Baselines
The naive reference point is end-to-end outcome checking: a judge sees only the final state and the task, and every step of a failed run inherits the failure. AgentEval reports failure-detection recall of 0.41 for this kind of evaluation on its text workflows. We expect a similar floor on MMInA. For online use, the naive reference is the unassisted agent. MMInA reports 21.77% task success for GPT-4V against a 96.25% human baseline.
The strongest existing method is AgentEval-style step-level judging with a frontier multimodal model given full trajectory history, which is how one would naively extend AgentEval to screenshots. AgentEval reports recall of 0.89 on its internal workflows, at least 0.78 on T-bench and SWE-bench, and 72% root-cause accuracy against an 81% human ceiling, all text-only. We found no reported result on Lacuna for step-level judging on MMInA or any multimodal trajectory benchmark, so we will produce this number ourselves. For fairness, the frontier judge gets the same rubric, the same step segmentation, and the same prompt-tuning budget on a shared development split as our judge. We will report its cost per trajectory. Beating it means higher step-level recall at equal or better precision, higher root-cause accuracy, and a false-acceptance curve that stays flat over step index while the frontier curve rises. If the frontier judge wins on accuracy, the result counts only as a cost win, and we will say so.
The strongest compact competitor is MR. Judge-7B used zero-shot as a local judge. MR. Judge reports 75.5% overall and 83.2% on hallucination on VL-RewardBench, against 65.8% overall for GPT-4o. It shares our backbone, so it isolates the value of transition-specific training data. Two ablations isolate the key ingredients. The first is the frontier judge and the untrained 7B backbone, each run under the local condition. This separates the effect of context isolation from the effect of specialization. The second varies the counterfactual taxonomy, removing one falsification type at a time, to show which training negatives matter. All 7B variants use the same training compute and data volume.
Why not just…?
Why not just use a frontier judge with the full trajectory, as Mind2Web 2 does? This is the most serious threat. Mind2Web 2 shows 99.03% agreement with humans for a frontier agent-as-judge. However, its checks are leaf-level verifications of final answers, each grounded in a specific cached page. That design already localizes the evidence, which is consistent with our hypothesis rather than against it. The measurement study settles the question directly. If the frontier full-history judge shows no rise in false acceptance with step index on MMInA, the inherited-hallucination claim is wrong. The contribution would then shrink to the cost reduction, which still matters for per-step online use given the cost gaps documented in LLMs Still Can't Plan; Can LRMs?: $42.12 versus $0.44 per 100 instances for two frontier models.
Why not just caption each screenshot and let a text judge run AgentEval? MLLM-as-a-Judge found that judges sometimes do better from descriptions than from pixels, so this could work. Against it, MMInA reports that agents with visual input outperform those limited to accessibility trees. Also, the errors we target, such as a wrong button or an unchanged page, are exactly the details a generic caption tends to smooth over. We run caption-then-judge as a full baseline under both the local and full-history conditions, with the same captioner for every step. If it matches the pixel judge, we will adopt it and report that.
Why not just window the context of an off-the-shelf judge, with no training? If context isolation is the whole story, a prompt change beats a trained model, and the ablation above tests exactly this. We expect windowing alone to remove the step-index trend but to leave a flat, high false-acceptance rate on subtle transitions. The reason is that general judges are tuned on static image–response pairs, not change detection. If the untrained local judge matches ours within noise, we will report the finding as primarily a measurement and design result.
Why not just re-weight attention toward image tokens inside the full-history judge, as ViF or SHARP do? Visual Multi-Agent System cuts error propagation by 30–40% with attention re-allocation. SHARP lowers CHAIR sentence-level hallucination (the share of image captions that mention absent objects) on LLaVA-v1.5 from 52.8 to 34.8 with steering vectors. The research file notes, however, that steering vectors learned on static images degrade under the distribution shift of multi-step rollouts. More importantly, neither method removes the agent's false narration from context. We apply ViF to the full-history frontier-open-weight judge as a baseline to test whether rebalancing attention is enough while the narration remains.
Proposed Experiments and Decisive Checks
The first decisive check is the false-acceptance-versus-step-index curve on annotated MMInA trajectories, for each judge under full-history, history-without-claims, and local conditions. A rising full-history curve that flattens under the local condition is the central evidence. If the curve rises under history-without-claims as well, the cause is visual context length. If it rises only with claims present, the cause is sycophancy toward the agent's narration.
The second check is step-level failure-detection recall and precision, and root-cause accuracy, against the baselines above. This is run on MMInA websites held out from training data and repeated on Mind2Web 2 trajectories. As a text control, the same pipeline is run on ALFWorld traces, where no visual grounding is involved; locality should neither help nor hurt much there, which guards against the result being a generic artifact of shorter prompts.
The third check is reasoning length. We compare no reasoning, capped reasoning, and uncapped reasoning in the local judge, to see whether the step-by-step-reasoning failure appears once history is removed.
The fourth check is online use. We add the judge to an open agent on MMInA and report task success, hop success, and the rate of wandering episodes, compared with the unassisted agent and with a frontier judge in the same loop. We report tokens and dollars per trajectory for each.
The fifth check is forgetting. Following the 4B degradation reported in BaT, we re-evaluate the specialized judge on VL-RewardBench and POPE to confirm that narrow training did not erase its general judging and hallucination-detection ability.
References
- Why LVLMs Are More Prone to Hallucinations in Longer Responses: The Role of Context
- Visual Multi-Agent System: Mitigating Hallucination Snowballing via Visual Flow
- AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- BaT: Towards Self-Evolving Medical Research Agent with Stage Rubrics
- MLLM-as-a-Judge: Assessing Multimodal LLM-as-a-Judge with Vision-Language Benchmark
- MR. Judge: Multimodal Reasoner as a Judge
- Flex-Judge: Text-Only Reasoning Unleashes Zero-Shot Multimodal Evaluators
- MMInA: Benchmarking Multihop Multimodal Internet Agents
- Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge
- Adaptive Safety Probing for Resource-Efficient Vision-Language-Action Models
- LLMs Still Can't Plan; Can LRMs? A Preliminary Evaluation of OpenAI's o1 on PlanBench
- SHARP: Steering Hallucination in LVLMs via Representation Engineering