This paper, recently presented at the NeurIPS 2026 Workshop on Interpreting Agent Behavior, moves beyond simple "win/loss" metrics for AI agents. Instead, it "instruments" the decision-making process, scoring language model (LM) agents at every single turn in complex games to see if their actions actually align with their stated beliefs.
TL;DR
Researchers developed a framework to evaluate LM agents in imperfect-information games by comparing their decisions and stated beliefs against game-theoretic "perfect" benchmarks (CFR+). They discovered a significant "belief-action gap": agents often fail to act consistently with the beliefs they explicitly state, and frequently fall into "state-blind" behavior where they ignore the game state entirely.
What Problem Does This Paper Address?
Current agent evaluations usually focus on task success—did the agent win the game or solve the puzzle? However, success can be a "lucky" outcome of a bad decision. In high-stakes fields like clinical triage or finance, we need to know if an agent's decision-making process is sound.
A major challenge is the faithfulness of reasoning. If an agent provides a Chain-of-Thought (CoT) explanation, is that actually why it made the choice, or is it just post-hoc justification? This paper tackles this by using zero-sum imperfect-information games (like Poker), where the "optimal" belief and action are mathematically computable, providing a ground-truth reference that most real-world tasks lack.
Methods
The authors placed three mid-sized LMs (12B–14B parameters) into five different games. At every decision node, the system performed two separate calls to the LM:
- The Action Call: "What move do you make?"
- The Belief Call: "What do you think the opponent’s hidden state (e.g., their card) is?"
They evaluated these outputs using eight properties derived from decision and game theory, including:
- Belief Error (P7a): How far the stated belief is from the "true" Bayesian posterior (using Total Variation Distance).
- Belief-Action Consistency (P7b): Does the action taken actually make sense given the stated belief?
- State-Blindness: A diagnostic check to see if the model's action changes at all when the hidden state changes.
The "gold standard" reference used is CFR+ (Counterfactual Regret Minimization), a standard algorithm for finding Nash equilibria in imperfect-information games.
Datasets
The study utilized the OpenSpiel framework to implement five zero-sum games:
- Kuhn Poker & Leduc Poker: Classic benchmarks for strategic betting and bluffing.
- Goofspiel: A card game involving simultaneous moves.
- Liar’s Dice: Tests the ability to update beliefs across a bidding sequence.
- Dark Hex (2x2): A version of Hex with partial visibility, used to measure the "Value of Information."
The models evaluated were Qwen2.5-14B-Instruct, OLMo-2-13B-Instruct, and Mistral-Nemo-Instruct-2407.
Key Insights and Ideas
- The "Reflex" Problem: At temperature 0 (greedy decoding), models are often "state-blind." In many scenarios, the model took the exact same action regardless of what it "knew" about the opponent's hidden state. It acts more like a static reflex than a reasoning agent.
- Beliefs are "Verbal Reports," Not Drivers: The models often stated beliefs that were somewhat accurate, but their actions didn't follow. The agreement between action and belief was only marginally better than random chance (and in OLMo's case, sometimes worse).
- Memory Breaks the Cycle: Adding a "memory" (a summary of previous hands) significantly reduced state-blindness. It seems to "nudge" the model out of repetitive, fixed responses.
- Framing Inconsistency: Because the "Action" and "Belief" calls are separate, the model might frame the opponent differently in each, leading to behavioral contradictions that wouldn't exist in a unified reasoning architecture.
Key Results
- Belief Accuracy: Stated beliefs were consistently 0.250 to 0.288 (Total Variation Distance) away from the equilibrium posterior. For context, they only closed about 16-29% of the gap between a "uniform guesser" and a "perfect Bayesian."
- Consistency Gap: Where the action actually responded to the state, the margin of agreement with the stated belief over chance was only 0.19 for Qwen and 0.17 for Mistral. OLMo scored -0.04, meaning its actions were less consistent with its beliefs than a random player would be.
- Temperature Sensitivity: Increasing temperature from 0 to 0.2 drastically reduced the number of "state-blind" cells (from 16 down to 5), suggesting that greedy decoding is a major bottleneck for agentic behavior.
Limitations and Future Work
- Model Size: The study was limited to the 12B-14B tier. The authors are currently scaling this to 30B+ models and reasoning-specific models (like the o1 family).
- Prompt Sensitivity: The results are tied to a specific prompt format; different prompts might yield different levels of consistency.
- Mechanistic Next Steps: The authors plan to move from behavioral observation to mechanistic interpretability, using linear probes to see if the "true" belief exists in the model's internal activations even when the "stated" belief is wrong.
- Intervention: The ultimate goal is to use these per-node scores to fine-tune models specifically for better decision-making and belief-action alignment.
Summary updated