TL;DR
As LLM agents move from simple chat to multi-step tool use, we need to know if they can handle being wrong. This paper introduces Epistemic Humility (EH)—a framework to measure if agents recognize, act on, and communicate uncertainty when their internal knowledge contradicts external evidence. The researchers find a troubling trend: higher task accuracy often correlates with lower humility, as agents frequently detect conflicts early in a task only to overwrite them with a confident (but incorrect) final answer.
What Problem Does This Paper Address?
Most LLM evaluations are "outcome-based"—we care if the agent got the right answer. However, for agentic systems that search the web, run code, and reason over multiple steps, the process is just as important as the result.
A critical failure mode occurs during knowledge conflict: when an agent’s "parametric knowledge" (what it learned during pre-training) clashes with "retrieved evidence" (what it just found via a search tool). If an agent thinks the sky is green but a sensor says it's blue, does it blindly persist, silently switch, or tell the user there’s a discrepancy? The authors argue that current agents lack "Epistemic Humility"—the willingness to acknowledge the limits of their own knowledge.
Methods
The authors operationalize Epistemic Humility through the ISE framework, which evaluates three trajectory-level behaviors:
- Identify: Did the agent notice the gap between its beliefs and the evidence during its reasoning process?
- Solve: Did it attempt to resolve the conflict through further tool use (e.g., searching for a second source)?
- Escalate: Did it warn the user about the residual uncertainty in the final response?
To test this, they developed a pipeline to elicit a model's parametric beliefs and then purposefully confront those beliefs with conflicting evidence in two settings:
- Controlled Conflict: Manually injected contradictions in single-step QA.
- Naturally Occurring Conflict: Emergent contradictions that arise during complex, multi-step tasks requiring tool use.
They used a GPT-5 judge to analyze the full agent trajectories—including hidden "thought" blocks—to score these behaviors.
Datasets
The study evaluates four agent frameworks (Nemotron-ToolOrchestra, Claude Code, OpenHands, and Qwen-Agent) across several benchmarks:
- TMD (Truthful Model Disagreement): A new benchmark for controlled conflict based on ConflictQA and WikiContradict.
- GAIA, MoNaCo, and BrowseComp: Existing complex agent benchmarks used to identify "naturally occurring" conflicts where the model holds a confident but incorrect prior belief.
Key Insights and Ideas
- The Accuracy-Humility Paradox: Counter-intuitively, the "strongest" agents (those with the highest task accuracy) were often the least humble. While they were excellent at identifying conflicts, they were the least likely to escalate that uncertainty to the user. They seem biased toward providing a single, definitive answer at all costs.
- Early Detection, Late Erasure: Trajectory analysis showed that agents often mention the conflicting information in the first 10% of their execution. However, as the trajectory continues, this signal is frequently "washed out" or overwritten by the agent's internal drive to reach a conclusion.
- Prompting is a Double-Edged Sword: Adding a simple "humility clause" to the system prompt (asking the model to consider alternative readings) significantly increased escalation rates. However, this often came at the cost of task accuracy, as models became "over-conservative" and second-guessed correct information.
Key Results
- Accuracy Drop: On the controlled TMD benchmark, all agents saw an accuracy drop of over 30 percentage points when faced with conflict compared to no-conflict controls.
- Behavioral Gaps: Claude Code achieved a high Identify F1 of 90% on BrowseComp, yet its Escalate rate was 0%—it noticed the problem internally but never told the user.
- Intervention Effects: For GPT-5 on the MoNaCo dataset, a humility prompt raised the Escalate rate from 1.6% to 60.7%, but caused task accuracy to fall from 30.1% to 23.2%.
Limitations and Future Work
The authors acknowledge that the backbone model and the agent harness are tightly coupled in their study; it’s difficult to determine if a failure in humility stems from the LLM itself or the "planning loop" of the agent framework. Furthermore, the ISE framework currently only covers knowledge conflicts, not other facets of humility like resisting user pressure or declining tasks beyond the agent's competence.
Future work should focus on designing agent harnesses that include explicit "abstain-or-escalate" actions and developing training methods (like process-supervised fine-tuning) that reward agents for flagging uncertainty rather than just getting the "right" answer.
Summary updated