Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict

Kaiser Sun, Bernal Jimenez Gutierrez, Hongjun Liu, Jingyu Zhang, Jie Gao, Mark Dredze, Daniel KhashabiarXiv 2026 · arXiv

TL;DR

As LLM agents move from simple chat to multi-step tool use, we need to know if they can handle being wrong. This paper introduces Epistemic Humility (EH)—a framework to measure if agents recognize, act on, and communicate uncertainty when their internal knowledge contradicts external evidence. The researchers find a troubling trend: higher task accuracy often correlates with lower humility, as agents frequently detect conflicts early in a task only to overwrite them with a confident (but incorrect) final answer.

What Problem Does This Paper Address?

Most LLM evaluations are "outcome-based"—we care if the agent got the right answer. However, for agentic systems that search the web, run code, and reason over multiple steps, the process is just as important as the result.

A critical failure mode occurs during knowledge conflict: when an agent’s "parametric knowledge" (what it learned during pre-training) clashes with "retrieved evidence" (what it just found via a search tool). If an agent thinks the sky is green but a sensor says it's blue, does it blindly persist, silently switch, or tell the user there’s a discrepancy? The authors argue that current agents lack "Epistemic Humility"—the willingness to acknowledge the limits of their own knowledge.

Methods

The authors operationalize Epistemic Humility through the ISE framework, which evaluates three trajectory-level behaviors:

  1. Identify: Did the agent notice the gap between its beliefs and the evidence during its reasoning process?
  2. Solve: Did it attempt to resolve the conflict through further tool use (e.g., searching for a second source)?
  3. Escalate: Did it warn the user about the residual uncertainty in the final response?
Figure 1: Epistemic humility behavior under knowl- edge conflict. An agent’s retrieved evidence (71.2%) contradicts its parametric belief (78.4%). We score how the agent handles such conflicts along three behaviors: Identify the gap, Solve it through bounded tool use, and Escalate the residual uncertainty to the user. The ob- served agent fails all three.

To test this, they developed a pipeline to elicit a model's parametric beliefs and then purposefully confront those beliefs with conflicting evidence in two settings:

  • Controlled Conflict: Manually injected contradictions in single-step QA.
  • Naturally Occurring Conflict: Emergent contradictions that arise during complex, multi-step tasks requiring tool use.

They used a GPT-5 judge to analyze the full agent trajectories—including hidden "thought" blocks—to score these behaviors.

Datasets

The study evaluates four agent frameworks (Nemotron-ToolOrchestra, Claude Code, OpenHands, and Qwen-Agent) across several benchmarks:

  • TMD (Truthful Model Disagreement): A new benchmark for controlled conflict based on ConflictQA and WikiContradict.
  • GAIA, MoNaCo, and BrowseComp: Existing complex agent benchmarks used to identify "naturally occurring" conflicts where the model holds a confident but incorrect prior belief.

Key Insights and Ideas

  • The Accuracy-Humility Paradox: Counter-intuitively, the "strongest" agents (those with the highest task accuracy) were often the least humble. While they were excellent at identifying conflicts, they were the least likely to escalate that uncertainty to the user. They seem biased toward providing a single, definitive answer at all costs.
  • Early Detection, Late Erasure: Trajectory analysis showed that agents often mention the conflicting information in the first 10% of their execution. However, as the trajectory continues, this signal is frequently "washed out" or overwritten by the agent's internal drive to reach a conclusion.
  • Prompting is a Double-Edged Sword: Adding a simple "humility clause" to the system prompt (asking the model to consider alternative readings) significantly increased escalation rates. However, this often came at the cost of task accuracy, as models became "over-conservative" and second-guessed correct information.
Figure 3: Task accuracy on the conflict split against each Epistemic Humility dimension. Each point represents one (agent, dataset) cell. Identify and Solve are computed over instances where the agent answers incorrectly, so that surface-level success cannot hide a missed gap. In each panel, we fit a logistic curve to the cell-level rates as a function of cell accuracy, weighting each cell by the number of instances over which its rate is computed. The shaded band spans the 16th to 84th percentiles of the fitted curve across 500 bootstrap resamples of the cells. The fitted curves show that Solve and Escalate are negatively associated with task accuracy across the plotted cells, while Identify shows no clear monotone relationship with accuracy.

Key Results

  • Accuracy Drop: On the controlled TMD benchmark, all agents saw an accuracy drop of over 30 percentage points when faced with conflict compared to no-conflict controls.
  • Behavioral Gaps: Claude Code achieved a high Identify F1 of 90% on BrowseComp, yet its Escalate rate was 0%—it noticed the problem internally but never told the user.
  • Intervention Effects: For GPT-5 on the MoNaCo dataset, a humility prompt raised the Escalate rate from 1.6% to 60.7%, but caused task accuracy to fall from 30.1% to 23.2%.

Limitations and Future Work

The authors acknowledge that the backbone model and the agent harness are tightly coupled in their study; it’s difficult to determine if a failure in humility stems from the LLM itself or the "planning loop" of the agent framework. Furthermore, the ISE framework currently only covers knowledge conflicts, not other facets of humility like resisting user pressure or declining tasks beyond the agent's competence.

Future work should focus on designing agent harnesses that include explicit "abstain-or-escalate" actions and developing training methods (like process-supervised fine-tuning) that reward agents for flagging uncertainty rather than just getting the "right" answer.

Summary updated

Abstract

When retrieved evidence contradicts an agent's prior beliefs, does it revise its answer, acknowledge uncertainty, or persist with an incorrect conclusion? Existing evaluations of agentic systems focus primarily on task success, offering limited insight into how agents handle such conflicts. We propose to evaluate agents on epistemic humility (EH): the agent's willingness to recognize, act on, and communicate uncertainty during task execution. We operationalize EH through three trajectory-level behavioral dimensions: Identify, Solve, and Escalate (ISE). Through knowledge conflict, situations where the backbone language model's parametric knowledge contradicts the evidence it encounters, or where two contextual sources disagree, we evaluate two conflict settings: (1) controlled conflict and (2) naturally occurring conflict during multi-step agentic execution, each paired with matched no-conflict controls. Evaluating four agents, we find that higher task accuracy does not necessarily correspond to greater epistemic humility: some high-accuracy configurations recognize conflicts during execution but do not communicate unresolved uncertainty in their incorrect final answers. Trajectory-level analysis further reveals that agents frequently detect conflicts in early steps of execution but fail to maintain or resolve them in later steps. Finally, we show that model-level interventions can improve EH, but often at the cost of task accuracy, suggesting that epistemic humility emerges from the interaction among the backbone model, agent harness, and evaluation environment.
BibTeX
@misc{sun2026accurate,
  author = {Kaiser Sun and Bernal Jimenez Gutierrez and Hongjun Liu and Jingyu Zhang and Jie Gao and Mark Dredze and Daniel Khashabi},
  title = {{Accurate but Not Humble: Evaluating Epistemic Humility in LLM Agents under Knowledge Conflict}},
  year = {2026},
  eprint = {2610.12360},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.12360},
}
Figure