Many agents now act in a hybrid action space: they can call a high-level primitive, such as an API function or a parameterized robot skill, or fall back to low-level atomic actions, such as mouse clicks or raw motor commands. The fallback exists for good reasons. It lets the agent fix states that a skill left slightly wrong, and it covers behaviors the library lacks. A separate line of work shows that language-driven agents are easily distracted by irrelevant text in their prompt.
Put the two together and a failure mode appears that neither literature measures. When the prompt is noisy, a hybrid agent may not fail outright. It may quietly abandon its efficient tools and grind through the task with slow atomic actions. Success rate barely moves, so the damage shows up only as wasted steps, and it becomes a hard failure only when the step budget runs out.
We propose to measure this "distraction-induced fallback" causally, by pairing each noisy prompt with a clean one at the same state. We then train the agent's choice between tool and atomic action to stay the same when irrelevant context is added, while still responding when relevant context changes. The claim to test is specific. An invariance penalty on the tool-versus-GUI choice, trained on paired clean and perturbed contexts, should reduce distraction-induced fallback and step count more than three simpler fixes, and it should generalize to distractor styles never seen in training. The three fixes are filtering the prompt, retrieving fewer tools, and simply training on noisy data.
Core Claim
The motivating evidence comes from the file's two digital hybrid benchmarks. OSWorld-MCP gives computer-use agents 158 tools alongside ordinary GUI actions, with distractor tools present. Tools help when used: Claude 4 Sonnet rises from 38.9% to 45.0% success under a 50-step limit. Yet the highest tool invocation rate any model reached was 33.3%, which the authors call a "GUI bias." ComputerRL shows why this matters: its combined API-plus-GUI agent needs roughly one third of the steps of a GUI-only agent.
What OSWorld-MCP cannot say is why the agent falls back. The fallback could be necessary, because no suitable tool exists. It could be a stylistic habit. Or it could be caused by distracting context. Skill Use or Skill Theater? shows why asking the model does not help. In text-only reasoning, models claim near-100% skill use while causal reliance on the skill sits below 20%, and judges that read reasoning traces cannot detect the gap. Only interventions reveal what drove a decision. We carry that interventional logic into hybrid action allocation.
We also connect it to evidence that distraction is worst when noise resembles the task. Bring the Apple, Not the Sofa finds that semantically similar irrelevant text drops OpenVLA on LIBERO-Goal from 77.5% to 18.9%. Tools that resemble the right tool should be the most damaging distractors.
The central quantity is distraction-induced fallback. Take a state $s$ where the agent, given clean context $c$, chooses a tool call. Now give it a perturbed context $c'$ that adds only task-irrelevant material, such as extra tool definitions or conversational filler. Distraction-induced fallback is the probability that the agent switches to a GUI action:
$$\mathrm{DIF} = P\big(\text{type}(a \mid s, c') = \text{GUI} \;\big|\; \text{type}(a \mid s, c) = \text{tool}\big).$$
Here $\text{type}(\cdot)$ is the action category, tool or GUI. We will report DIF next to its mirror image, the rate of GUI-to-tool flips, so that net-zero "churn" is visible. Churn means many decisions flip in both directions while accuracy looks unchanged, as the Skill Theater paper observed.
The proposed training change adds one term to ordinary reinforcement learning on the hybrid space:
$$\mathcal{L} = \mathcal{L}_{\text{RL}} + \lambda \, \mathbb{E}_{s, c, c'}\Big[\mathrm{KL}\big(\bar{p}(\text{type} \mid s, c) \,\|\, p(\text{type} \mid s, c')\big)\Big].$$
The symbols are as follows. $\mathcal{L}_{\text{RL}}$ is the ComputerRL policy objective. $p(\text{type}\mid s,c)$ is the probability, summed over the tokens that start each action, that the next action is a tool call rather than a GUI action. $\bar{p}$ is the same quantity on the clean context with gradients stopped. $\lambda$ weights the penalty.
The penalty covers only the tool-versus-GUI decision, not the full action. It is applied only to irrelevant perturbations. For relevant perturbations, such as deleting the tool the task needs, we add no penalty and in fact expect the choice to change. This distinguishes the proposal from simply making the agent insensitive to its prompt.
Research Program
Build the benchmark. The first piece of work is a paired-perturbation extension of OSWorld-MCP. For each task we generate five kinds of irrelevant perturbation. Three enlarge the toolset: extra tools from other applications, extra tools whose names and descriptions are semantically close to the correct one, and near-duplicate tools with non-functional signatures. The other two add instruction filler: human-style paraphrase with filler, and semantically similar irrelevant clauses in the style of Bring the Apple.
Build the relevance controls. We also generate two relevant perturbations: removing the correct tool, and making the correct tool return an error. Here a good agent should fall back, which lets us test whether the invariance penalty damages necessary fallback.
Replay trajectories. Paired evaluation is done by replaying recorded trajectories. At each state along a clean-context trajectory we query the policy under clean and perturbed contexts. This gives DIF and churn at matched states, rather than only comparing whole episodes that drift apart.
Train. The second piece is the training change above, built on the released ComputerRL codebase, starting from its 9B-scale open agent. The training perturbations are drawn from three of the five irrelevant families. The remaining two, near-duplicate decoys and semantically similar filler, are held out for evaluation. The held-out split is the test that separates learned invariance from memorized noise patterns.
Transfer check. The third piece checks whether the effect holds when the macro primitives are skills the agent writes for itself rather than a fixed library. On AppWorld, SAGE lets agents either call raw APIs, the atomic level, or save and reuse composite Python functions, the macro level. It reports trouble retrieving skills across scenarios as the library grows. We will grow the saved-skill library with skills from unrelated scenarios, measure the rate at which the agent drops back to raw APIs at matched states, and apply the same invariance penalty to the choice between saved skill and raw API.
Why not robotics. We deliberately do not make robotics the main testbed. The robot hybrid methods, MAPLE, RAPS, and HRLP, choose primitives from state vectors or clean images and receive no language context. Adding linguistic noise to them would test a system that does not exist yet. Their lesson, that atomic fallback is valuable for gluing skills together, instead constrains our design: the penalty must not suppress necessary fallback.
Baselines
Do nothing. The naive reference is the ComputerRL agent evaluated with the full perturbed context and no filtering. It tells us how large DIF and step inflation are before any fix.
Strongest existing agent. The strongest trained competitor is ComputerRL's reinforcement learning on the API-plus-GUI space itself, which reports 48.9% success on OSWorld. That figure is on plain OSWorld rather than OSWorld-MCP. The file gives no reported result for any trained open agent on OSWorld-MCP with distractor tools, and no reported per-condition result isolating the with-distractor versus without-distractor gap. The closest reported reference on OSWorld-MCP is Claude 4 Sonnet at 45.0% under the 50-step limit, together with the 33.3% ceiling on tool invocation rate. We will rerun the ComputerRL agent on our paired split with the same starting checkpoint, the same number of environment rollouts, and the same step limits as our method. Beating it means lower DIF and fewer average completion steps on the held-out perturbation families, at equal or higher success, with no loss of the correct fallback rate on the relevance controls.
Tool retrieval. The OSWorld-MCP retrieval baseline shows the model only top-ranked tools. The stronger version is a SkillFlow-style pipeline of dense retrieval, two rounds of reranking, and final selection by a language model. SkillFlow reports raising Pass@1 on SkillsBench from 9.2% to 16.4%.
Instruction filtering. A small upstream model rewrites the instruction to remove filler. This follows Bring the Apple, which reports that such a filter recovers about 98.5% of clean performance for vision-language-action models. All filters receive the same compute for development, and their thresholds are tuned on the training perturbation families, as ours is.
Augmentation-only ablation. The ablation that isolates the key ingredient is identical RL training on the same perturbed contexts but without the paired invariance term. If the gain comes merely from seeing noise during training, this ablation will match our method. A second ablation applies the penalty to the entire action distribution rather than only the tool-versus-GUI choice. This tests whether restricting it to the action type matters.
Why not just…?
Why not just filter the toolset with retrieval? This is the most serious threat for toolset bloat, and it may win on the easy distractor families. The evidence that it falls short elsewhere is mixed rather than decisive. OSWorld-MCP used retrieval and still observed the 33.3% invocation ceiling. SkillFlow reports that its final language-model selection step is conservative and discards valid domain-specific skills. Retrieval also does nothing for filler in the instruction itself. The paired experiment settles the question directly. If retrieval matches our method on DIF and steps for the semantically similar decoy tools, which retrieval should find hardest to reject, we will report that and narrow our claim to instruction noise and combined noise. We will also evaluate retrieval and invariance combined, since they are not exclusive.
Why not just clean the instruction upstream with a small model? Bring the Apple's recovery figure is strong, but it was measured on single-shot robot commands with no toolset in context. A text filter also sees neither the tool list nor the screen state, so it cannot remove distraction coming from tools. Our evaluation runs this filter on both families of noise. We expect it to help on filler and not on tool distraction, and we report whichever pattern appears.
Why not just train with noisy data? Plain augmentation is the natural reviewer fix. It is exactly our main ablation. Our hypothesis is that augmentation reduces DIF on the noise styles seen in training but not on the held-out ones, because nothing in the reward ties the choice at a given state across contexts. If augmentation matches the invariance penalty on held-out families, the penalty adds nothing and we will say so.
Why not just instruct the agent to prefer tools? A prompt telling the agent to prefer tools will likely raise the invocation rate. Skill Theater shows, however, that prompt-driven skill mentions need not reflect causal use. A blanket preference would also damage the relevance controls, where falling back is correct. We include this prompt as a baseline and score it on both DIF and the correct fallback rate.
Proposed Experiments
Is the problem real? The first experiment establishes whether distraction-induced fallback exists and how large it is. Using the paired OSWorld-MCP split and the do-nothing agent, we measure four things per perturbation family: DIF, the reverse flip rate, average completion steps, and success under both the 15-step and 50-step limits. The prediction that would make the project worthwhile has three parts. DIF should be substantial. It should be larger for semantically similar decoys than for tools from other applications. And it should cost success mainly under the tight 15-step limit while costing mostly steps under the loose one. If DIF is near zero, the GUI bias is a habit rather than distraction, and the invariance method is unmotivated.
Does the method work? The second experiment compares our method with all baselines on the held-out perturbation families. It uses the same starting checkpoint, rollout budget, and tuning budget throughout. The decisive check has two conditions. First, lower DIF and fewer steps than the augmentation-only ablation, at no loss of success. Second, on the relevance controls, where the correct tool is removed or broken, a switch to GUI action at least as reliable as the base agent's. A method that lowers DIF by becoming insensitive to relevant context fails this check.
Does it transfer to self-written skills? The third experiment repeats the paired measurement on AppWorld as the saved-skill library grows with skills from unrelated scenarios. We report Task and Scenario Goal Completion together with the rate of fallback to raw APIs. SAGE's reported 72.0% Task Goal Completion and 60.7% Scenario Goal Completion on Test-Normal serve as the reference for the uninflated library. The question is whether the method limits the loss as irrelevant skills accumulate.
Is fallback a symptom of distraction? The final analysis asks whether DIF predicts episode-level step inflation across tasks. If it does, a hybrid agent's retreat to atomic actions can be used as a cheap, measurable signal of prompt distraction, one that its reasoning trace would never reveal.
References
- OSWorld-MCP: Benchmarking MCP Tool Invocation In Computer-Use Agents
- ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents
- Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents
- Bring the Apple, Not the Sofa: Impact of Irrelevant Context in Embodied AI Commands on VLA Models
- Reinforcement Learning for Self-Improving Agent with Skill Library
- Augmenting Reinforcement Learning with Behavior Primitives for Diverse Manipulation Tasks
- Accelerating Robotic Reinforcement Learning via Parameterized Action Primitives
- Hierarchical Reinforcement Learning with Parameters
- SkillFlow: Scalable and Efficient Agent Skill Retrieval System