TL;DR
Standard Mechanistic Interpretability (MI) tools can detect and even steer LLM agents away from failures like web element hallucinations, but they struggle to provide a complete causal explanation. This paper demonstrates that while we can predict agent failures before they happen, the underlying "circuits" are too distributed and entangled with prompt formatting for current MI methods to fully decode.
What Problem Does This Paper Address?
As LLM-based agents are deployed for complex, long-horizon tasks like web navigation, they frequently fail in ways that differ from standard text generation. One common failure is web element hallucination: the agent tries to click or type into a button or search bar that simply doesn't exist on the page.
While Mechanistic Interpretability (MI) has successfully mapped internal "circuits" for simple tasks (like factual recall or grammar), applying these tools to agents is a different beast. Agents are sequential and interact with dynamic environments, creating a "moving target" for analysis. The authors investigate whether standard MI tools—like linear probing, activation patching, and steering—can actually explain why these agentic failures occur.
Methods
The researchers analyzed Qwen2.5-7B-Instruct as it performed web navigation tasks. They employed a suite of MI techniques:
- Linear Probing: Training simple classifiers on the model's internal activations to see if they could predict a hallucination before it was generated.
- Representation Steering: Using the weights from those probes to "nudge" the model's activations during inference, attempting to flip a hallucinated action into a valid one.
- Activation Patching: Swapping activations between "clean" (valid) and "corrupted" (hallucinated) runs to identify specific causal components.
- Logit Lens: Projecting intermediate layer activations back into vocabulary space to see how the model's "intent" (e.g., the desire to "Search") evolves across layers.
- Sparse Autoencoders (SAEs): Using pre-trained SAEs to see if they could identify interpretable "features" responsible for the failure.
Datasets
The study utilized Online-Mind2Web, a benchmark for web navigation on live websites. The authors constructed a dataset of 300 episodes and 2,071 actions, where approximately 32.6% of the actions were hallucinations (e.g., the model proposing a "Search" action when no search bar was present).
Key Insights and Ideas
- Hallucinations are "Pre-meditated": Linear probes could predict a hallucination with an AUC of 0.80 at the very last token of the prompt—before the model even started generating the action. This suggests the commitment to a failed strategy happens early in the residual stream.
- The Localizability Trap: Standard MI often assumes a "circuit" is a small, localized set of neurons or heads. In agents, this breaks down. The authors found that computation is highly distributed; intervening on single components fixed at most 8.9% of failures.
- Feature Entanglement: Probes are brittle. A minor change in prompt formatting (e.g., switching from a JSON action list to a numbered markdown list) caused probe performance to plummet. This indicates the "hallucination" signal is deeply entangled with the specific way information is presented to the agent.
- Stationarity Violations: In agents, the environment changes at every step. This makes "minimal pair" counterfactuals (a staple of MI) nearly impossible to build, as removing one action causes the entire future trajectory to diverge, making the original and counterfactual runs incomparable.
Key Results
- Steering Works (Surface Level): Steering the model along the probe direction corrected 85% of hallucinations, though it disrupted 7% of valid actions.
- Distributed Failure: As shown in Figure 2, most Feed-Forward Network (FFN) layers and attention heads have only diffuse, minor effects on grounding. No single "smoking gun" component was responsible for the hallucinations.
- Format Sensitivity: When the prompt format was shifted (OOD), the steering fix rate dropped by ~27 percentage points, highlighting that the identified "hallucination direction" is not universal.
Limitations and Future Work
The authors conclude that current MI tools are "model-centric" but need to become "agent-centric." * Data Mismatch: Existing Sparse Autoencoders (SAEs) are trained on general text and fail to capture agent-specific concepts like "goal tracking" or "tool selection." * Need for New Benchmarks: The field lacks MI benchmarks specifically for agents that include ground-truth causal triggers for failures. * Beyond Single Timesteps: Future work must develop methods to track how plans and beliefs emerge and evolve across long trajectories and multiple environment interactions, rather than looking at isolated tokens.
Summary updated