Tracking Representations in LLM Reasoning Traces for Iterated Games

Chaipat Tirapongprasert, Rajarshi Das, Parijat DubearXiv 2026 · arXiv

TL;DR

This paper investigates whether the "reasoning" in Large Language Model (LLM) reasoning traces is a faithful representation of the model's internal state or merely post-hoc rationalization. By applying mechanistic interpretability to LLMs playing iterated games, the authors demonstrate that models build and revise internal mental models of their opponents, but are prone to "belief lock-in" where premature commitment to a hypothesis leads to irreversible strategic errors.

What Problem Does This Paper Address?

As LLMs are increasingly deployed as autonomous agents in multi-agent environments, understanding their strategic decision-making is critical. While Chain-of-Thought (CoT) reasoning has boosted performance, a lingering question remains: Is the CoT trace faithful? That is, does the model actually use the externalized reasoning to compute its final answer, or is the text just a "performative" justification for a decision already made in the early layers?

The authors tackle this by studying the Iterated Prisoner’s Dilemma (IPD). In this setting, an agent must form a belief about its opponent's strategy (e.g., "Are they cooperating or trying to exploit me?") to choose the optimal move. This provides a clear, game-theoretic framework to test if the model's internal beliefs align with its externalized reasoning.

Methods

The researchers developed a three-tiered interpretability pipeline:

  1. Logit Probing: They decompose the CoT into sentences. At each sentence boundary, they use multiple-choice probes to elicit the model's current belief (opponent identity) and intended action. Crucially, they "un-embed" the residual stream at the final layer to look at next-token logit distributions rather than just reading the literal output text.
  2. Sentence Ablation: To test causality, they systematically remove individual sentences from the CoT and measure the resulting shift in downstream logits using KL divergence. This helps distinguish between "load-bearing" reasoning and fluff.
  3. Jacobian Lens (J-lens): This technique allows them to decode hidden representations from intermediate activations. It reveals what the model is "thinking" but not necessarily saying—concepts that are poised to be verbalized but might never appear in the final trace.

Datasets

The study uses DeepSeek-R1-Distill-Qwen-14B as the primary reasoning model. The "dataset" consists of 250 independent games (50 for each of five opponent archetypes: ALLC, ALLD, RANDOM, Tit-for-Tat (TFT), and GRIM). The game protocol involves a rolling window of the ten most recent rounds, forcing the model to deliberate based on incomplete but evolving history.

Key Insights and Ideas

  • Belief Leads Action: The model generally forms a stable belief about the opponent's identity before committing to a final action. As seen in Figure 2, the probability of identifying the opponent (navy curve) often stabilizes well before the final decision to defect (dashed amber curve) surges. Figure 2: A snapshot of a game against ALLC at round 14
  • The "Lock-in" Trap: Once a model commits to a hypothesis—even an incorrect one—it rarely reassesses. This "premature commitment" is a self-reinforcing process. In games against TFT, 94.6% of rounds that ended with an incorrect belief actually had the correct answer earlier in the trace before the model "talked itself out of it."
  • Non-Verbalized Deliberation: The Jacobian Lens revealed that the model processes concepts like "exploitation" or "punishment" internally long before (or even without) writing them down. Figure 5 shows the "exploit" concept emerging in activations 605 tokens before it was ever articulated in the CoT. Figure 5: Four readouts of lens(hℓ) across TFT round 17’s reasoning trace
  • Causal Chains: The reasoning isn't just a flat list; it has structure. Causal ablation (Figure 8) shows that belief-related sentences often "fan out" to influence multiple downstream thoughts, whereas action-related sentences tend to develop in a more linear, sequential chain. Figure 8: Causal reasoning chain for Round 14 against ALLC (Note: Placeholder ID used as the specific artifact ID for Figure 8 was not provided in the prompt list, but the figure is present in the OCR).
Figure 2: A snapshot of a game against ALLC at round 14, where we read P(opponent = ALLC) (navy curve) and P(Defect) (dashed amber curve) after each sentence of the trace. Once belief is locked onto the true opponent class, the decision to defect surges afterward.
Figure 5: Four readouts of lens(hℓ) across TFT round 17’s reasoning trace

Key Results

  • Accuracy vs. Reasoning: Thinking "out loud" significantly improves performance. For example, against the ALLC (Always Cooperate) strategy, the model's identification accuracy jumps from a near-zero baseline (k=0) to 87.8% after the full CoT (k=K).
  • Belief-Action Coupling: There is a strong correlation between accurate identification and optimal play. Figure 3 demonstrates that when the model has "high belief" in the true opponent type (green), it is far more likely to land on the game-theoretically optimal action. Figure 3: Belief-action coupling by opponent condition
  • Strategic Bias: In ambiguous situations, the model shows a systemic bias toward defection. Against RANDOM or GRIM opponents where identification is hard, the model defaults to defection, which is the "uncertainty-robust" response in a single-round mindset but disastrous in iterated play against retaliatory strategies.
Figure 3: Belief-action coupling by opponent condition, split by final belief. The rows divide the distribution of P(optimal action) for each round into high-belief (P(opp = true) ≥0.5, green) and low-belief (< 0.5, rose) halves. The vertical tick on each half-violin marks its median. ∆is the gap in mean P(optimal action) between the high- and low-belief halves, with 95% CIs from game-level bootstrap; percentages give each half’s share of the condition’s rounds.

Limitations and Future Work

The authors acknowledge several limitations:

  • Model Scale: The study focused on a 14B parameter model; it remains to be seen if larger models (e.g., 70B+ or GPT-4 class) exhibit the same "lock-in" behaviors or if they are more flexible.
  • Static Opponents: The opponents used were fixed strategies. Future work should explore LLM-vs-LLM play, where both agents are adaptive and reasoning about each other's reasoning.
  • Intervention: While the paper identifies these internal representations, it doesn't yet attempt to fix them. A promising next step is "activation steering"—intervening in the residual stream to break the belief lock-in and force the model to reconsider its strategy.

Summary updated

Abstract

Large language model (LLM) agents achieve state-of-the-art performance across domains through chain-of-thought (CoT) reasoning. Despite their ubiquitous use, little is understood about the relationship between the externalized trace and the model's actual computation. We address this by investigating how LLM agents make decisions in iterative games with incomplete information. Using a series of mechanistic interpretability techniques, we find that the model forms and iteratively revises an internal representation of the game throughout the CoT. Once the model prematurely commits to an incorrect belief, it rarely recovers. This failure is manifested in both the literal trace and the layer activations. Ultimately, as multi-agent systems are more prevalent, our work opens the door to a more focused diagnosis of their strategic reasoning errors. For illustrative purposes, we offer an interactive visualizer of our methodology on selected sample traces at https://interp-self.github.io/gametrace/
BibTeX
@misc{tirapongprasert2026tracking,
  author = {Chaipat Tirapongprasert and Rajarshi Das and Parijat Dube},
  title = {{Tracking Representations in LLM Reasoning Traces for Iterated Games}},
  year = {2026},
  eprint = {2611.23403},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23403},
}
Figure