This paper uncovers Procedural Exposure Bias (PEB), a systematic flaw where language model agents over-favor actions simply because they appear frequently in their provided "skill library," regardless of their actual relevance. The authors demonstrate that presenting these skills visually (as images) rather than as text significantly mitigates this bias, leading to more robust decision-making and higher task success across multiple benchmarks.
What Problem Does This Paper Address?
To solve complex, long-horizon tasks, modern AI agents are often "skill-augmented"—they are given a library of reusable procedures (e.g., "how to use a microwave") to guide their actions. While this makes knowledge accessible, it doesn't guarantee the model will use it correctly.
The authors identify a fundamental issue: Large Language Models (LLMs) are highly sensitive to the frequency of patterns in their input context. If a certain action is mentioned repeatedly in a skill description, the model starts to "prefer" that action, even if the current environment state makes it inapplicable. This is Procedural Exposure Bias (PEB). The core challenge is finding a way to provide this procedural knowledge without inducing such surface-level statistical biases.
Methods
The researchers developed a controlled probing framework to isolate PEB. They sampled decision steps from successful agent trajectories and introduced "distractor" actions—actions that are task-relevant but currently inapplicable. By varying how many times a distractor's keyword appeared in the skill library (the "exposure level"), they could measure how much the model's preference shifted purely due to repetition.
A key novelty is the comparison between Text and Visual skill representations:
- Text Skill: The procedure is provided as standard language tokens.
- Visual Skill: The same text is rendered into an image (a "skill card") and processed through a Vision-Language Model's (VLM) vision encoder.
- Rasterized Condition: To prove the benefit wasn't just about fancy layouts, they tested a "rasterized" version—plain text turned into a simple image—to isolate the effect of the visual encoding pathway itself.
They also performed layer-wise readouts and activation patching (intervening in the model's internal computations) to understand where and how this bias manifests within the neural network's architecture.
Datasets
The agents were evaluated on three diverse interactive benchmarks:
- ALFWorld: A household simulator for embodied tasks (e.g., "put the apple in the sink").
- ScienceWorld: A complex environment for performing elementary science experiments.
- WebShop: A simulated online shopping environment requiring product search and attribute selection.
Key Insights and Ideas
- Repetition is a Trap: In textual representations, simply repeating a keyword like "microwave" in the background skills makes the model significantly more likely to try to "open the microwave," even if it's standing in front of a fridge.
- Visual Encoding as a Filter: Visual representation acts as a natural "decoupler." Because visual encoding transforms text into distributed representations over image patches rather than discrete lexical tokens, it breaks the direct link between word frequency and action preference.
- Signal vs. Evidence: Interestingly, the repeated information is still recoverable from the visual representation (the model "knows" it's there), but it doesn't translate into "action evidence" as strongly as text does.
- Bias is a Late-Stage Phenomenon: Layer-wise analysis showed that PEB isn't present in the early layers of the model; it develops progressively and becomes most pronounced toward the output layers.
Key Results
- Higher Success Rates: Across all four evaluated backbones (Qwen3.5-9B/27B, Gemma-4-31B-it, and Kimi-K2.5), using Visual Skills led to higher macro-average success rates compared to Text Skills.
- Bias Attenuation: Visual representation substantially reduced the preference shift induced by repeated distractor keywords. As shown in Figure 1, the "distractor action probability" climbs steeply with exposure in Text but remains much flatter in Visual.
- Visual Encoding > Layout: The "Rasterized" experiments confirmed that the majority of the benefit comes from the visual encoding process itself, with structured layout providing only a marginal additional improvement.
Limitations and Future Work
While the paper clearly demonstrates that visual encoding helps, it doesn't fully pinpoint the exact mechanistic reason why the VLM's vision-to-language bottleneck filters out frequency bias so effectively. The authors also note that the response to internal component interventions (like MLP patching) is architecture-specific, suggesting that different models might "process" this bias through different internal pathways.
Future work could explore more sophisticated visual formats or investigate if similar "decoupling" effects can be achieved within purely textual models through different training objectives or architectural changes.
Summary updated