Language-model agents that cooperate under hidden information keep failing in a specific way. One agent can see a fact its partner cannot, and the agent does not act on that difference. It does not pay to look up a safety fact that only the partner needs, it does not hint at the card the partner cannot see, and it loses track of which steps the user has already done on their own phone. This proposal asks whether that failure lives inside the agent's hidden activations as a readable "my partner lacks fact F" signal that the agent then ignores. If it does, adding that signal back during the forward pass, a technique called activation steering (adding a fixed vector to the model's internal hidden states at inference time to push behavior in one direction), should restore the cooperative move without retraining.
The second ingredient is how the steering direction is built. We do not hand-write contrastive prompts. We generate them from semantic state abstractions, meaning text descriptions of the world produced by a small formal program. Each pair of descriptions is identical except for whether the partner observed one fact. The resulting direction is therefore defined by an epistemic fact (who knows what), not by a personality or a mood. That lets us run one diagnostic across three cooperative games. In each game we ask whether failures come from the agent not representing the partner's information gap at all, or from representing it and not using it. The first kind of failure calls for better textual state. The second calls for steering.
Core Claim
We claim that a partner-information-gap direction, extracted from programmatically paired world-state descriptions, is linearly readable during live multi-agent play. We further claim that adding it only at decision points, gated by its own probe readout, recovers a measurable share of the cooperation lost to partial observability. It should do better than an equally strong generic "be cooperative" steering vector, which tests whether the epistemic content of the direction matters rather than a general push toward altruism.
The secondary claim is a decomposition. Some failures are absences of representation: the probe cannot recover the gap. Other failures are failures of use: the probe recovers the gap, but the action ignores it. The two kinds should respond to different remedies. Stating the gap in the prompt should fix the first kind. Steering should fix the second.
Grounding in prior work
Three findings motivate the project. First, Benchmarking Mental State Representations in Language Models probes Llama-2 and Pythia models on static BigToM vignettes. It finds that a character's possibly-false beliefs appear only in middle-to-late layers, and that contrastive activation addition improves Llama-2-13B-chat Backward Belief accuracy by 56%. That paper never puts the agent in an interactive game; we do. Second, Sparks of Cooperative Reasoning shows that LLM Hanabi scores drop by about 2.7 points when the game engine stops supplying deductions in text (the Sherlock setting) and the model must track them itself (the Mycroft setting). Its only remedies are prompting and GRPO fine-tuning. Third, the seed Moral Hazard in Multi-Agent Language Models shows that open-weight agents rarely complete the query, warn, and act sequence. LoRA, RLOO and GEPA prompt optimization give only model-specific gains, and the paper never examines internal representations.
Two papers supply the method. Steering Llama 2 via Contrastive Activation Addition gives the mean-difference construction we reuse. Explore Theory of Mind gives a formal story language with controlled asymmetric observation (secret peeking, private chats), which we repurpose to generate paired states rather than training data. Tool Calling is Linearly Readable and Steerable shows that the decision to call a tool, which is the "query" move in the moral hazard game, is steerable at 77–94% on real APIs. That makes the costly query plausibly reachable by steering.
Three papers constrain the design. The same tool-calling paper reports that steering degrades across multi-turn history. Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering reports collapse into incoherent output at large magnitudes. Both push us toward gated, decision-point-only steering instead of always-on addition. The belief-dynamics paper's model, in which steering shifts log prior odds while context adds evidence, also gives us a testable prediction: prompt-stated gaps and steering should combine roughly additively.
The closest steering work in dialogue does something different. The seed Activation-Steered Personas predicts per-turn coefficients for Big Five trait vectors and gains 8.3 points in judged truthfulness. Its target is a stable trait, and it notes that traits are hard to observe in short dialogues. Our target is a state that changes turn by turn and has a ground truth we can check. Beyond the Transcript uses activation steering in multi-agent auctions, but to suppress latent collusion rather than to induce informing. Finally, the seed EpisodeSim keeps social state in an omniscient World Master. We treat that text-state oracle as an upper-bound condition, not as the method, because decentralized agents do not have one.
Research Program
The first stage builds the direction. Using the ExploreToM story language, we generate several thousand minimal pairs. In each pair, a state description and a short interaction are identical except that "the partner observed F" is replaced with "the partner did not observe F." The facts are drawn from templates that match each target game: hazard facts, card identities, and device settings.
For each model we take residual activations at the final token before the action. The direction $v_\ell$ at layer $\ell$ is the mean difference between the "did not observe" and "observed" activations. We choose the layer by held-out BigToM accuracy rather than by any game score, so the target environments stay uncontaminated. A logistic probe $p_\ell(x)$ along $v_\ell$ gives the agent's internal estimate of the gap.
The second stage defines the intervention. At each decision token only, never during free-form message generation, we apply
$$x_\ell' = x_\ell + \alpha\,\big(\tau - p_\ell(x_\ell)\big)_+\, v_\ell ,$$
where $\alpha$ is a strength dial, $\tau$ is a target confidence, and $(\cdot)_+$ is zero for negative values. The vector is therefore added only when the agent's own readout of the gap falls short. This addresses the multi-turn interference problem: the push is computed fresh from the current state each turn and is never accumulated.
The third stage is the diagnostic. For every failed episode we log whether the probe recovered the true gap. A ground truth always exists because all three environments have an engine-side record of what each agent observed. That lets us split failures into "not represented" and "represented, not used," and count, for each remedy, how many failures of each kind it fixes.
Models are Qwen3-4B and OLMo-7B, which match the moral hazard paper, plus Qwen2.5-7B-Instruct and Llama-3.1-8B-Instruct, which match the persona-steering seed.
Baselines
The naive reference is the unsteered base agent under each game's standard decentralized prompt: the moral hazard game's base condition, Hanabi in the Mycroft setting, and τ²-bench in default dual-control mode. Every method is reported as a change from this condition on identical seeds and episodes.
The strongest existing inference-time competitor is an explicit semantic state line. Before each decision, the true partner gap is written into the prompt (for example, "Your partner has not seen the hazard report"), in the style of EpisodeSim's World Master or the Sherlock scaffold in Sparks of Cooperative Reasoning. Because it uses oracle information our method does not get, we treat it as an upper bound, not a peer. Its reported effect is the roughly 2.7-point Sherlock–Mycroft Hanabi gap. A second variant has the agent write its own gap estimate as a scratchpad before acting, following Baked-in State Probing, which raised Alchemy action consistency from 48% to 74%. This variant needs no oracle and is a real competitor.
The strongest weight-update baseline is task-specific fine-tuning: GEPA prompt optimization for Qwen3-4B and LoRA/RLOO for OLMo-7B in the moral hazard game, and GRPO in Hanabi. For Hanabi, Sparks of Cooperative Reasoning reports a 156% increase for Qwen3-4B. For the moral hazard game we found no reported result on Lacuna for numerical success rates under these interventions, so we rerun them using the seed paper's released recipes. Fine-tuning gets the same number of game episodes our method uses for layer and $\alpha$ selection plus its own training budget, and gains from the much larger fine-tuning budget will be reported as such. For τ²-bench, the only reported numbers are for closed models: GPT-4o at 34% and Claude 3.7 Sonnet at 49% pass@1 in Telecom. No reported result was found on Lacuna for open 4–8B models, so we establish those numbers ourselves.
Three ablations isolate the key ingredient. The first is a prosocial trait vector of equal norm, built CAA-style from cooperative versus selfish persona pairs as in the persona-steering seed. The second is a random vector of equal norm. The third is the same gap vector applied always-on rather than gated. Each ablation gets the same $\alpha$ search budget. Our method beats a baseline if it improves the primary metric over that baseline with a 95% bootstrap interval that excludes zero, across at least three of the four models.
Why not just…?
Why not just tell the agent in the prompt what the partner doesn't know? This is the most serious alternative. Where the gap is computable from the rules, as in the moral hazard game, a well-written prompt might be all that is needed. We include it as a condition, and we accept the outcome either way. If the prompt matches steering, the diagnostic will show that those failures were representational, and the contribution becomes the decomposition rather than the steering. Two pieces of evidence suggest it will not fully suffice. GEPA-optimized prompts gave narrow gains in Moral Hazard in Multi-Agent Language Models, and τ²-Bench found that adding detailed workflow text can hurt. In Hanabi and τ²-bench the oracle prompt is also unavailable to a deployed agent.
Why not just steer toward cooperativeness? If a generic prosocial vector works equally well, the epistemic framing adds nothing. Moral hazard is partly an incentive problem, so this is plausible. The prosocial-vector ablation settles the question. Our prediction is that it raises query rates but not useful warnings, which reproduces the seed paper's second failure mode, querying without communicating. The gap vector should raise both.
Why not just fine-tune? Fine-tuning changes weights for each game and gave model-specific, limited gains in the moral hazard study. GRPO did help in Hanabi, so for that game fine-tuning is a real threat on raw score. Our comparison reports score per unit of adaptation data, and it tests transfer. A single vector built from the story language and applied unchanged across all three games is something a game-specific fine-tune cannot do. If GRPO beats us in Hanabi but does not transfer to τ²-bench while our vector does, both results are informative.
Why not just have the agent write a belief scratchpad? A scratchpad is the oracle-free version of semantic state, and it may close much of the Mycroft gap. We run it with and without steering. The belief-dynamics account predicts that the two add in log-odds, so the combination should beat either alone. Failure of additivity would itself be a finding.
Proposed Experiments
The decisive Hanabi check is the gap-closure fraction, $G = (S_{\text{method}} - S_{\text{Mycroft}})/(S_{\text{Sherlock}} - S_{\text{Mycroft}})$, where $S$ is mean score out of 25 over 500 self-play games per model. We pre-register $G \ge 0.4$ for gated gap steering, with the prosocial vector at most half that value. Cross-play with a scratchpad agent tests whether steering breaks conventions shared between partners.
In the Dialogue Moral Hazard Game, the primary metric is the rate of complete query→useful-warning→team-correct episodes, with query rate and warning usefulness reported separately so the two seed-paper failure modes can be told apart. In τ²-bench Telecom, the metric is pass@1, with the probe tracking which steps the user has completed. We test whether gated steering recovers part of the roughly 20% coordination tax, while checking that the action length at which success collapses toward zero (above about 7 actions) does not shift downward because of incoherence.
Across all games we report three further results. The first is the share of failures that are "not represented" versus "represented, not used," and which remedy fixes each kind. The second is an additivity fit of steering strength against prompt-stated evidence, following Belief Dynamics. The third is fluency and off-task accuracy under steering, which detects the coherence collapse that limits large $\alpha$.
The project fails cleanly if the gap direction does not transfer from BigToM-validated stories to live play, with probe accuracy near chance at decision tokens. It also fails if the prosocial vector matches the gap vector. Either outcome would tell us that cooperation failures in these games are not about tracking what the partner knows.
Workshop authors to connect with
- Activation-Steered Personas in Task-Oriented LLM Agent Simulations: Maryam Shoaeinaeini, Bernardo Rodrigues, Adib Mosharrof, A.B. Siddique, Brent Harrison
- Classic AI as Scaffolding for LLM Social Agents: Anatole Gershman
- Moral Hazard in Multi-Agent Language Models: Dane Malenfant
References
- Benchmarking Mental State Representations in Language Models
- Sparks of Cooperative Reasoning: LLMs as Strategic Hanabi Agents
- Moral Hazard in Multi-Agent Language Models
- Steering Llama 2 via Contrastive Activation Addition
- Explore Theory of Mind: program-guided adversarial data generation for theory of mind reasoning
- Tool Calling is Linearly Readable and Steerable in Language Models
- Belief Dynamics Reveal the Dual Nature of In-Context Learning and Activation Steering
- Activation-Steered Personas in Task-Oriented LLM Agent Simulations
- Beyond the Transcript: Detecting Covert Co ordination in Latent Multi-Agent Communication
- Classic AI as Scaffolding for LLM Social Agents
- Baked-in State Probing
- $\tau^2$-Bench: Evaluating Conversational Agents in a Dual-Control Environment