Represented but Not Used: Value-Policy Dissociation in LLM Agents

Yuzhang Luo, Mohamed Aghzal, Liangming Pan, Ziyu YaoarXiv 2026 · arXiv

TL;DR

Researchers have discovered a "value-policy dissociation" in LLM agents: models often internally represent the correct value of future actions even when their final policy chooses a suboptimal one. By identifying specific "Natural Value Transport" (NVT) directions in hidden states, the authors developed a steering mechanism that successfully bridges this gap, significantly improving agent performance in navigation and web-shopping tasks.

What Problem Does This Paper Address?

Large Language Model (LLM) agents are increasingly used for sequential decision-making, yet they frequently fail at simple planning tasks—often falling into "loops" where they repeat the same incorrect actions.

The central question is: Why do they fail? Is it a "perception" failure (the model doesn't understand the environment or the consequences of its actions) or a "decision" failure (the model understands the consequences but fails to act on that information)?

Previous work in mechanistic interpretability has shown that LLMs represent world states (like the layout of a board game). This paper shifts the focus to action value—the expected future reward of a specific move. The authors investigate whether LLMs internally "know" which move is better, even when they choose the worse one.

Methods

The authors analyzed three models—Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B—using a three-step approach:

  1. Probing for Value: They trained linear probes to decode the "action value" (e.g., distance to a goal or expected shopping reward) from the models' intermediate hidden states.
  2. Quantifying Dissociation: They measured how often the internal "value probe" correctly identified the best action while the model's actual "policy" (the output token) chose a worse one.
  3. Intervention via Natural Value Transport (NVT): To fix the dissociation, they moved beyond simple "steering" (pushing activations along the probe direction). Instead, they introduced NVT, a method that learns how hidden states naturally change when action values vary. They then used this as an "actuator" to steer the model toward higher-value actions during inference.
Figure 1: Value–policy dissociation from a motivating example to a systematic pattern. Top: a looping Grid trajectory in which the agent repeats an inferior action despite observations favoring a better alternative. Bottom: Generalizing to broader scenarios, a linear readout of the hidden state favors the better action while the policy selects the worse one; this probe–policy mismatch recurs across states.

Datasets

The study utilized two distinct environments:

  • Grid: A 5x5 partially observed navigation task where the agent must reach a goal while remembering obstacles seen in previous steps.
  • WebShop: A complex, simulated e-commerce environment where agents must navigate search results and product pages to find items matching a specific instruction.

Key Insights and Ideas

  • The "Knowing-Doing" Gap is Real: In the Grid environment, for up to 58% of states where the model made a mistake, the internal value probe actually knew the correct action. This confirms that the failure isn't just about "not seeing" the right path; it's about failing to use that internal knowledge.
  • Value is Represented Early: Action value information is linearly decodable quite early in the model's layers, often peaking in the middle-to-late layers before the final output.
  • Sensors $\neq$ Actuators: A major finding is that the direction you use to read information (the probe) is not necessarily the best direction to change behavior. Direct steering along probe directions had inconsistent results. However, NVT—which captures how the model's internal state naturally shifts between "good" and "bad" outcomes—proved to be a much more effective "actuator" for policy control.
Figure 2: Layer-wise value readout in Grid and WebShop. (a) Grid held-out top-1 ranking accuracy, measuring whether the action with highest decoded value matches the gold best-value action; the dashed line shows the Manhattan baseline. (b) WebShop held-out pairwise ranking accuracy, measuring whether the readout correctly orders two candidate actions by empirical continuation value; dashed lines show the TF-IDF baselines. Shaded regions show 95% cluster-bootstrap intervals for the learned readouts. Results reported on a held-out validation set.

Key Results

The authors implemented a closed-loop controller using the value probe as a "sensor" and NVT as an "actuator." This intervention led to significant performance gains:

  • WebShop Success: For Qwen2.5-32B-Instruct, the mean reward increased from 0.358 to 0.452 (a ~26% improvement).
  • Grid Navigation: Success rates improved across the board. For example, Qwen2.5-7B-Instruct's success rate jumped from 14% to 30%, while its "loop rate" (repetitive mistakes) dropped from 85% to 64%.
  • Late-Layer Sensitivity: The steering was most effective when applied to later layers, identifying a specific "policy-relevant" region where internal representations are translated into final choices.
Figure 3: Steering slope results. NVT (blue) shows a more consistent and coherent ability to influence the model's policy in later layers compared to standard probing or random directions.

Limitations and Future Work

  • White-Box Requirement: The current method requires access to internal activations and gradients, making it inapplicable to closed-source models like GPT-4 or Claude.
  • Task-Specific Supervision: Fitting the NVT operator requires task-specific data (e.g., knowing the "gold" distance to a goal), which may not be available for all real-world tasks.
  • Linearity Assumption: The study focuses on linear readouts and transport operators. Future work could explore whether non-linear dynamics play a role in how value information is propagated.
  • Generalization: It remains to be seen if these specific "value directions" generalize across different types of tasks or if they must be relearned for every new environment.

Summary updated

Abstract

Large language model (LLM) agents need to choose actions whose consequences unfold over multiple future steps, yet poor decisions do not reveal whether the model failed to represent those consequences or failed to use them. Mechanistic interpretability has studied internal representations of environment state and plans, but sequential decision making also requires evaluating how candidate actions change future outcomes. Reinforcement learning formalizes this as action value. We study this computation in Qwen2.5-7B-Instruct, Qwen2.5-32B-Instruct, and GPT-OSS-20B across a partially observed Grid environment and WebShop, a simulated web shopping environment. We find that the relative values of candidate actions are linearly decodable from intermediate hidden states, including at states where the policy prefers a worse action. We call this gap between value representation and policy choice value-policy dissociation. To investigate how represented value relates to action selection, we compare direct interventions along the value readout direction with Natural Value Transport (NVT), a transformation learned from hidden state changes that naturally result from differences in action value. While readout-direction interventions have limited and inconsistent effects, NVT more consistently influences action preferences, particularly at later layers. We also test a closed-loop controller that uses the value readout as a sensor and NVT as an actuator, observing improvements in task outcomes. These results suggest that LLM agents can represent useful prospective value without reliably translating it into policy choice.
BibTeX
@misc{luo2026represented,
  author = {Yuzhang Luo and Mohamed Aghzal and Liangming Pan and Ziyu Yao},
  title = {{Represented but Not Used: Value-Policy Dissociation in LLM Agents}},
  year = {2026},
  eprint = {2611.10457},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.10457},
}
Figure