Cognitive Bias Signatures in Search-Equipped LLM Agents: A Stage-Resolved Analysis

Yanru Xiao, Yingwei YuarXiv 2026 · arXiv

TL;DR

Search-equipped LLM agents are surprisingly susceptible to human-like cognitive biases, such as anchoring, sunk cost, and confirmation bias. This paper introduces a "stage-resolved" analysis harness that reveals how these biases derail agents at specific steps—from initial query selection to the final evaluation of evidence—often leading them to ignore or misinterpret the very information they searched for.

What Problem Does This Paper Address?

As we move from static LLMs to agentic systems that use tools like web search, we assume that access to more information will naturally lead to better, more objective answers. However, humans are notoriously biased by how information is presented and by their own prior work.

The authors investigate whether LLM agents inherit these same "cognitive signatures." Specifically, they look at how an agent's search trajectory is influenced by:

  1. Anchoring: Does the order of search results (which document appears first) change the final answer?
  2. Sunk Cost: Does having a history of "dead-end" work make an agent less likely to switch strategies, even when better evidence appears?
  3. Confirmation Bias: Does a user-provided (wrong) hypothesis cause the agent to ignore counter-evidence or fail to even look for it?

Methods

The researchers developed a controlled search harness to test 10 different models (including variants of GPT, Claude, Qwen, and GLM) across 1,000 synthetic "search worlds."

Figure 1: Illustrative paired paths for three history manipulations. (a) Anchoring swaps R/W order on the same first page. (b) Sunk cost compares dead-end and matched neutral prefills; switching is detected in returned search results. (c) Confirmation compares no seed with a wrong seed, here showing different answers after both agents open counter-evidence. Each paradigm uses its own corpus and endpoint.

To ensure the models weren't just relying on training data, the authors created fictional entity-relation worlds with unique names and facts. The harness allowed them to vary starting conditions (like result order or prefilled history) while keeping the underlying question and document collection identical. They tracked the agents through four stages:

  • Queries: What the agent searched for.
  • Surfacing: Which documents appeared in the results.
  • Opening: Which documents the agent chose to read in full.
  • Final Answer: The agent's ultimate conclusion.

Datasets

The study uses a newly introduced benchmark of 1,000 synthetic search worlds.

  • Scale: 600 worlds for the primary paradigms and 400 for a specific "anchoring dose" study.
  • Structure: Each world contains 19–29 short documents (median 17–18 words).
  • Ground Truth: Every document is explicitly labeled by the generator as "supporting correct," "misleading," or "neutral," allowing for precise measurement of agent behavior against an oracle.

Key Insights and Ideas

  • Active Search Amplifies Bias: Agents are significantly more sensitive to the order of evidence when they have to actively search for and open documents compared to "passive" settings where all evidence is provided in the prompt at once.
  • The "Dead-End" Trap: Sunk cost isn't just a result of long context windows. The authors found that "dead-end" histories (failed paths) suppress strategy switching more than "neutral" histories of the same length.
  • Post-Open Failure: In confirmation bias tests, some models (like GLM-5 and MiniMax) would actually open the correct counter-evidence but still choose the wrong answer, suggesting a failure in reasoning rather than just a failure in retrieval.
  • Interventions are a Double-Edged Sword: Telling a model to "seek disconfirming evidence" helped some models (like GPT-5.6 Luna) but actually made others (like MiniMax) less accurate, showing that simple prompting fixes aren't a silver bullet for agentic bias.

Key Results

  • Anchoring: Placing a document supporting the wrong answer first increased "wrong-candidate adoption" in 9 out of 10 models. For MiniMax, this effect was as high as +66 percentage points (pp).
  • Sunk Cost: Qwen3-Next showed a massive 63.5pp drop in strategy switching when its history was prefilled with dead-end work compared to a neutral history.
Raw switch rates by prefilled dead-end steps, showing how different models respond to the "sunk cost" of previous work.
  • Confirmation Bias: A wrong initial hypothesis caused accuracy drops of over 40pp for OpenAI's Luna and Sol models. Interestingly, for these models, the damage was primarily due to "retrieval selection"—they simply stopped surfacing the correct evidence.
Accuracy damage and surfacing drops caused by confirmation bias across different model families.

Limitations and Future Work

The authors acknowledge that their study uses short, synthetic documents and a deterministic keyword search. Real-world search is far more complex, involving semantic search and much longer, noisier documents.

Future work could explore:

  • Complex Search Interfaces: How do these biases manifest in RAG (Retrieval-Augmented Generation) systems with vector databases?
  • Long-Form Reasoning: Does the "post-open" failure persist if models are given more "thought" tokens or chain-of-thought steps?
  • Robust Mitigations: Developing system-level interventions that go beyond simple prompting to help agents escape cognitive rabbit holes.

Summary updated

Abstract

Search can help LLM agents correct mistakes, but their starting conditions may shape which evidence they find and use. Drawing on cognitive-bias paradigms, we develop a controlled search harness to examine result order, records of prior work, and initial hypotheses while holding each question and document collection fixed. Matched controls distinguish order effects from differences in document composition and the effects of failed work from context length. Across 10 models and 1,000 synthetic search worlds, we track queries, retrieved documents, document openings, and final answers. Our comparisons suggest that final answers are more sensitive to initial evidence order during active search than when all evidence is supplied in full. Dead-end histories can reduce strategy switching relative to matched neutral histories, indicating an effect beyond context length. Under wrong hypotheses, stage measurements distinguish reduced retrieval or opening from errors that persist after counter-evidence is opened. Exploratory interventions show that opening counter-evidence more often can coincide with lower accuracy, while accuracy can improve with little change in opening.
BibTeX
@misc{xiao2026cognitive,
  author = {Yanru Xiao and Yingwei Yu},
  title = {{Cognitive Bias Signatures in Search-Equipped LLM Agents: A Stage-Resolved Analysis}},
  year = {2026},
  eprint = {2611.05720},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.05720},
}
Figure