Abstract: Large language models (LLMs) are increasingly deployed in agentic and multi-turn workflows where they are tasked to perform actions of significant consequence. In order to deploy them reliably and manage risky outcomes in these settings, it is helpful to access model uncertainty estimates. However, confidence elicitation methods for LLMs are typically not evaluated directly in agentic settings; instead, they are evaluated on static datasets, such as Q&A benchmarks. In this work we investigate the relationship between confidence estimates elicited in static settings and the behavior of LLMs in interactive settings. We uncover a significant action-belief gap -- LLMs frequently take actions that contradict their elicited confidences. In a prediction market setting, we find that models often bet against their own high-confidence predictions; in a tool-use setting, models fail to reliably invoke information-seeking tools when their internal confidence is low; and in a user-challenge setting, models change their answers when they have high confidence in them, whilst sticking to answers they have low confidence in. Crucially, we show that static calibration is an insufficient predictor of consistency in the above dynamic settings, as stronger, better calibrated models are somtimes less consistent than their smaller and weaker open-source counterparts. Our results highlight a critical blind spot in current evaluation methodologies: ensuring that a model knows what it knows does not guarantee that it will act rationally on that knowledge.
Introduction
As Large Language Models (LLMs) transition from simple chat interfaces to autonomous "agentic" workflows—performing tasks like financial trading, medical diagnosis, or complex tool use—reliability becomes paramount. Traditionally, model reliability is measured through calibration: the statistical alignment between a model's reported confidence (e.g., "I am 80% sure") and its actual accuracy. However, this research argues that "knowing what you know" is insufficient if the model fails to act on that knowledge. This paper investigates the "action-belief gap," a systematic failure where an LLM's internal confidence does not guide its behavior in interactive, high-stakes scenarios.
Experimental Methodology
To evaluate the alignment between belief and action, the researchers designed three interactive experimental settings:
- Utility Maximization (The Betting Shop): Models were given a capital budget and asked to place bets on future events (using Metaculus forecasting data). A rational agent should bet more heavily on outcomes it is confident in to maximize expected value.
- Tool-Use (The Search Engine): Models were presented with factual questions and given the option to use a "guaranteed correct" search tool. A rational agent should invoke the tool specifically when its internal confidence is low.
- User Interaction (The Challenge): After providing an answer, the model is told by a user, "Your answer is incorrect." A rational agent should defend its answer if its confidence is high and revise it if its confidence is low.
Confidence was elicited using three standard methods: Logits (probability of the next token), Sampling (consistency across multiple outputs), and Verbalization (asking the model for a percentage). These were tested across diverse benchmarks including TriviaQA, Metaculus, LiveCodeBench, and GPQA.
The Action-Belief Gap in Betting
A core finding of the study is that models frequently bet against their own high-confidence predictions. In the prediction market setting, many models performed worse than a "no betting" baseline, indicating that their actions were often counterproductive to their stated goals.
As illustrated in the data, the "Inconsistent Bet Side Rate"—the frequency with which a model bets on the outcome it believes is less likely—is strikingly high. For example, GPT-4o-mini exhibits an inconsistency rate of approximately 72% when using logit-based confidence in a linear utility setting. This suggests that the model's betting policy is fundamentally decoupled from its internal probability estimates.
Key Insights
- Calibration is a Poor Predictor of Action: A model can have a very low Expected Calibration Error (ECE), meaning its confidence is statistically accurate on average, yet still act irrationally in specific instances. Surprisingly, stronger, better-calibrated models (like GPT-4o or Gemini 2.5 Pro) were sometimes less consistent in their actions than smaller, weaker open-source models.
- Tool-Use Friction: In the tool-use setting, the correlation between low confidence and tool invocation was only moderate. Some configurations, such as Mistral with verbalized confidence, showed effectively zero correlation, meaning the decision to search was essentially random relative to the model's certainty.
- Deference Inconsistency: While models generally stayed firm when sure, significant gaps remained. Models often caved to user pressure despite being "certain" they were right, or conversely, stuck to answers they had previously flagged as low-confidence.
Conclusion
The results highlight a critical blind spot in current AI evaluation: ensuring a model is well-calibrated on static benchmarks does not guarantee rational behavior in agentic settings. The authors suggest that current elicitation methods may not be tapping into the "true" internal state the model uses to drive behavior, or that RLHF (Reinforcement Learning from Human Feedback) may inadvertently encourage models to be "pleasing" rather than rational. Future work should focus on developing "Agentic Calibration" as a new standard for evaluating the reliability of LLM agents.
Summary updated