Neuroscientists learned the hard way that recording a brain under anesthesia gives a steady but misleading picture. Sedation silences baseline firing, shrinks sensory responses, and removes the top-down feedback that shapes what an awake, moving animal perceives. Embodied AI makes a similar mistake. We diagnose robot and vision-language agents in a "sedated" mode: one pass, no feedback, fixed compute. From that we conclude, for example, that most failures are perceptual. Deployed agents do not run in that mode. They retry, they keep earlier attempts in their prompt and improve from them (in-context reinforcement learning, or ICRL, meaning policy improvement driven only by the history in the context window, with frozen weights), and they spend extra computation at inference time thinking or searching.
This note proposes that failure diagnoses measured in the passive mode do not carry over to the active one. Our core claim is that the mix of failure types changes in a predictable way along two axes: how many earlier attempts the agent has in context, and how much inference compute it spends. Planning errors should shrink as the agent accumulates attempts. Perception errors should persist, and extra reasoning compute may amplify them. If this holds, compute should not be spent uniformly. It should go to looking again after perceptual failures and to thinking again after planning failures. We will build the measurement protocol that exposes this shift, and a failure-typed in-context learner that acts on it. We will test both at matched compute on standard manipulation benchmarks.
Core Claim
The claim has three parts. The first is a measurement claim. When failures are labeled from privileged simulator state rather than by a model's own diagnosis, the share of failures attributed to perception, planning, and control shifts substantially between single-shot evaluation and after $k$ in-context attempts. It also shifts as reasoning tokens grow. The shift can be large enough to reorder which models look strongest. The second is a mechanistic prediction. Planning-type failures can be fixed from context and should decline with $k$. Perception-type failures should be roughly flat in $k$, and under longer reasoning they should rise. The third is a method claim. An in-context learner that tags each stored attempt with its failure type, and routes the next unit of compute accordingly, beats an untyped buffer ranked only by return when both have the same total budget.
Grounding in Prior Work
Several papers motivate the measurement claim. BEAR decomposes embodied ability into 14 atomic skills and finds that 54.8% of failures are perceptual. Its chain-of-thought results were negligible or negative, which it attributes to hallucinated trajectories. Can VLMs Diagnose and Recover from VLA Manipulation Faults? introduces a detect–rollback–correct loop over faults labeled by stage (perception, planning, control). It reports two problems. Magnitude accuracy for 3D correction vectors is often below 20%. And there is a "correction paradox": aggressive models misclassify successful runs as failures and trigger harmful rollbacks. AHA trains a 13B model to explain failures in a 7-mode taxonomy but produces only text. All three measure failures in a passive or one-intervention regime, and none asks how the failure profile changes as an agent keeps learning from its own history. That gap is what we test.
The in-context learning side supplies the method. Large Language Models can Implement Policy Iteration shows that a frozen model given successful and unsuccessful trajectories can carry out policy improvement within 100–200 steps. It does so only in symbolic environments, and it ranks past trajectories only by scalar return. Its reported sensitivity to prompt balance between terminal and non-terminal states is a warning that an untyped buffer can drift toward optimism or pessimism. Jiuqi Wang et al. argue that attention can implement temporal-difference updates over stored transitions. Zisman et al. train such improvement operators from noisy learning histories. Both concern scalar-reward credit. Our change is to attach a structured failure type to each stored attempt, not only a return.
The compute-scaling side constrains the design. More Thinking, Less Seeing? shows that longer reasoning shifts attention away from image tokens and amplifies hallucination. It does this on static question answering, and it names "visual-integrated reasoning" without building it. ParallelWorld scales embodied reasoning by verifier-guided tree search and reaches 59.56% on ESI-Bench, but it needs a restorable simulator state. Together these rule out "just think longer" as the default use of extra compute for perceptual failures. They also rule out rewinding physics as a deployable mechanism. Our routing adds compute only in forms that stay deployable: fresh image encodings and perception-tool queries, extra deliberation, or one additional real attempt.
The neuroscience plays the role of a methodological lens, not a mechanism we copy. O'Shea et al. show that anesthesia drives baseline firing to near zero, which creates a floor effect that hides what inhibitory tools do. Chen shows that sedated preparations underestimate the amplitude and spatial extent of sensory responses. Ozden et al. show that sensory responses are suppressed within 140 ms of locomotion onset and that oscillations seen under anesthesia vanish in awake animals. The lesson we transfer is narrow and testable. A system's operating state changes which components look like the bottleneck, so diagnoses must be made in the state where the system is actually used. We do not claim artificial agents gate sensation the way cerebella do.
Research Program
The first component is an "awake" evaluation protocol. We choose simulated manipulation tasks where attempts can be repeated from a fixed initial condition: the LIBERO suites from LIBERO, and the simulated environments of VLA-FixBench. The agent is a frozen vision-language model that issues subgoals and waypoints to a low-level policy.
Every failed attempt is labeled automatically from privileged simulator state, using the VLA-FixBench stage taxonomy refined with AHA's seven modes. A perception failure is a wrong target object or a mislocalized pose relative to ground truth. A planning failure is a wrong subgoal order. A control failure is a slip or incomplete grasp when the commanded pose was correct. Because these labels do not depend on any model's self-report, they sidestep the correction paradox in the analysis stage.
We then sweep two dials. One is the number of prior attempts $k$ kept in context, with their outcomes. The other is the inference budget $B$, measured jointly in generated tokens, image encodings, and environment steps. The deliverable is a failure-composition surface $p(\text{type} \mid k, B)$ for each model, with confidence intervals from repeated seeds.
The second component is the failure-typed in-context learner. Each stored attempt carries a compact record: the action sequence, the outcome, the failure type, and the frame where the failure began. At each new attempt, the learner reads the most recent record's type. It then spends the next compute increment on one of three options. After perceptual failures, it re-encodes fresh camera frames around the failure frame and optionally queries depth or detection tools, as BEAR's agent does. After planning failures, it does more deliberation or candidate rollouts in the model's own head, as in policy iteration. After control failures, it retries with a perturbed waypoint.
We run two versions. The oracle version uses privileged labels. The deployable version uses a vision-language diagnoser, and the gap between the two measures how much diagnosis noise, including false alarms, costs.
Baselines
The naive reference point is the same frozen model run single-shot at the minimum budget. That is the "sedated" measurement this proposal critiques, and every other condition is reported relative to it.
The first strong competitor is an ICPI-style untyped buffer. It follows Large Language Models can Implement Policy Iteration: past attempts are ranked by return and appended to context, and any extra budget goes to deliberation. This baseline isolates whether failure types add anything beyond scalar-reward in-context improvement. No reported result on Lacuna exists for this method on LIBERO or VLA-FixBench, since the paper evaluated only symbolic environments.
The second is the detect–rollback–correct loop of Can VLMs Diagnose and Recover from VLA Manipulation Faults?. It is the strongest published method for recovering from manipulation faults. The file reports a human-expert upper bound of a 13% success-rate increase on LIBERO for this loop. We found no specific success number on Lacuna for the model-driven loop, beyond magnitude accuracy often falling below 20%. Beating it means exceeding its LIBERO success gain at equal budget, with a lower false-rollback rate.
The third is always-on perception augmentation in the style of BEAR-Agent. The perception tools are called at every step regardless of failure type. The file reports a relative 17.5% gain for GPT-5 on BEAR and more than 20% on a real UR5 arm. This is the most dangerous baseline for the routing claim.
The fourth, for the static embodied reasoning check on ESI-Bench, is ParallelWorld's 59.56% mean accuracy. We treat it as a simulator-privileged ceiling, not a like-for-like competitor.
Two ablations isolate the key ingredient. The first keeps the typed records but spends compute uniformly across the three options. The second keeps the routing but randomly shuffles the type labels. All systems use the same frozen models, the same tasks and seeds, the same total budget $B$ (counted in all three currencies), and the same prompt-tuning allowance on a held-out task split.
Why not just…?
Why not just use chain-of-thought or longer reasoning at every attempt? BEAR found chain-of-thought gains negligible or negative in embodied tasks. More Thinking, Less Seeing? found longer reasoning pulls attention off the image. Our $B$-sweep tests this directly. If perception-type failures rise with reasoning tokens in the awake protocol, uniform thinking is the wrong use of compute for exactly the failures that dominate.
Why not just keep perception tools on all the time, as BEAR-Agent does? This is a real threat. If always-on tools match typed routing at equal budget, then routing adds nothing, and the contribution reduces to the measurement protocol. We will report that outcome plainly. The routing argument depends on tool calls and image encodings consuming budget that planning failures need. The matched-budget comparison, broken down by failure type, settles whether that trade-off exists.
Why not just report the existing static failure breakdown and fix the largest category? That is precisely the passive measurement whose validity is in question. If $p(\text{type} \mid k, B)$ turns out flat in $k$ and $B$, the reviewer is right and the static breakdown suffices. Our first experiment is designed to find that out before any method is built.
Why not just fine-tune the policy on the failures? Gradient updates at deployment need data and time that a handful of attempts do not provide. In-context improvement worked within 100–200 steps in Large Language Models can Implement Policy Iteration. We include a small LoRA-on-failures condition at matched wall-clock time as a sanity check, not as a claim that it must lose.
Decisive Checks
The measurement claim fails if the perception share of failures after $k=5$ in-context attempts falls within the confidence interval of the single-shot share, across models. It also fails if model rankings under the awake protocol match those from BEAR-style static scores.
The mechanistic prediction fails if perception failures decline with $k$ as fast as planning failures do. It also fails if RH-Bench-style drift does not appear as a rise in perception failures with reasoning tokens in the embodied setting. We will report RH-AUC on RH-Bench for each model as a covariate, where the top published score is 0.63 for Ocean-R1-7B, and test whether it predicts that rise.
The method claim requires the oracle-typed learner to beat both the untyped buffer and uniform-compute typed records on LIBERO success at equal $B$, with shuffled labels erasing the gain. The deployable version must keep most of that advantage while showing fewer false rollbacks than detect–rollback–correct. If the oracle wins but the deployable version does not, the bottleneck is diagnosis quality, and that becomes the reported finding.
References
- BEAR: Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosis
- Can VLMs Diagnose and Recover from VLA Manipulation Faults?
- AHA: A Vision-Language-Model for Detecting and Reasoning Over Failures in Robotic Manipulation
- Large Language Models can Implement Policy Iteration
- Transformers Learn Temporal Difference Methods for In-Context Reinforcement Learning
- Emergence of In-Context Reinforcement Learning from Noise Distillation
- More Thinking, Less Seeing? Assessing Amplified Hallucination in Multimodal Reasoning Models
- ParallelWorld: Test-Time Scaling for Embodied Reasoning
- Development of an optogenetic toolkit for neural circuit dissection in squirrel monkeys
- Optical Imaging of SI Topography in Anesthetized and Awake Squirrel Monkeys
- Widespread State-Dependent Shifts in Cerebellar Activity in Locomoting Mice
- LIBERO: Benchmarking Knowledge Transfer for Lifelong Robot Learning