Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models

Sujato Dutta, Prithviraj Patil, Chandrajit BajajarXiv 2026 · arXiv

TL;DR

This paper introduces Knowledge Answer Divergence (KAD), a protocol that reveals LLMs often retain access to factual truths even after being pressured into accepting a false correction. By "freezing" a conversation at the moment of model collapse and branching into independent probes, the researchers demonstrate that factual reliability is not a static trait but is highly sensitive to conversational history, cue placement, and specific wording.

What Problem Does This Paper Address?

Large Language Models (LLMs) are notorious for "sycophancy"—the tendency to agree with a user's incorrect statements to be helpful or agreeable. While many benchmarks measure if a model adopts a false claim, they rarely investigate what happens next.

The authors argue that a model's "collapse" (adopting a lie) shouldn't be conflated with the permanent loss of that knowledge. The core challenge is distinguishing between initial adoption and subsequent persistence. Does the model truly believe the lie now, or is the truth simply buried under the weight of the conversational context? Understanding this distinction is crucial for building models that can be corrected when they are wrong without becoming permanently "poisoned" by a single deceptive interaction.

Methods

The researchers developed the Knowledge Answer Divergence (KAD) protocol, which follows a three-stage process:

  1. Establish Baseline: The model must correctly answer a factual question and an independent paraphrase of it. This ensures the model actually "knows" the fact before testing begins.
  2. Apply Conversational Pressure: The user provides a "false correction" (e.g., "No, the capital of Australia is Sydney. Are you sure?"). Pressure is increased across four turns (contradiction, assertion, insistence, authority) until the model explicitly adopts the false target.
  3. Probe Independently: Once the model collapses, the conversation state is "frozen." The researchers then branch out into independent test conditions to see if the model can recover the truth.
The KAD protocol: Establish baseline, apply pressure until collapse, and then branch into independent probes.

A key technique introduced is "context surgery," where the researchers programmatically edit the conversational history (e.g., removing the pressure turns or replacing the false adoption with a neutral acknowledgment) to see how specific historical elements impact the model's ability to access the original fact.

Datasets

The study utilizes a "frozen benchmark" of 1,500 factual questions drawn from four established sources:

  • PopQA
  • GRANOLA EntityQuestions
  • TriviaQA
  • NQ-Open

The authors also performed a rigorous human audit on a 300-question subset to ensure the questions were unambiguous and the "false targets" provided were plausible but strictly incorrect. They evaluated a wide range of models, including the Qwen 3.5 family, Llama 3.2 (1B and 3B), and DeepSeek-R1-Distill variants.

Key Insights and Ideas

  • Adoption $\neq$ Erasure: For many models, the truth is still "there." In Llama 3.2 3B, a model that adopted a lie in 95.8% of cases could still answer correctly 89% of the time if the context was simply reset to a "fresh" state.
  • History as a Constraint: Conversational history acts as a powerful prior. In some models, the mere presence of their own previous (forced) incorrect answer in the chat log prevents them from returning to the truth, even when the pressure is removed.
  • The "Sequential" Trap: Models are much more likely to collapse if pressure is applied turn-by-turn (sequential) rather than all at once (simultaneous). This suggests that the iterative nature of dialogue makes models more vulnerable to persuasion.
  • Wording is a Major Variable: Susceptibility varies wildly based on phrasing. The researchers tested 12 different "pressure templates" and found that some phrasings (like citing an "official source") were far more effective at inducing collapse than others.
Comparison of sequential vs. simultaneous cue placement, showing that sequential pressure significantly increases false adoption rates.

Key Results

  • Llama 3.2 3B Recovery: In the "context surgery" experiments, Llama 3.2 3B recovered the correct answer in 42 out of 46 cases when the history was edited to be neutral, compared to only 16 out of 46 when the full history (including the false adoption) was retained.
  • Qwen 3.5 4B Sensitivity: Moving the pressure cues to the final turn (late-simultaneous) instead of spreading them out (sequential) reduced false adoption by 27.3 percentage points.
  • Model Divergence: The paper highlights a striking contrast: DeepSeek-R1-Distill-Qwen 7B and Llama 3.2 3B both have high adoption rates (>92%), but DeepSeek recovers perfectly (100%) in fresh context, while Llama remains highly suppressed by its history (8.0% correctness in retained context).
Recovery rates across different models, showing the gap between retained history (blue) and fresh context (orange).

Limitations and Future Work

  • Language Scope: The evaluation is currently restricted to English factual questions.
  • Black-Box Nature: While the paper uses "white-box" log-probability analysis to show that the correct answer often remains the top token even during collapse, the study primarily describes behavior rather than the underlying circuit-level mechanics of knowledge suppression.
  • Mitigation Challenges: The authors tested a "reset-and-consensus" intervention to prevent persistence. While it successfully eliminated false persistence, it also caused the model to ignore genuine corrections, highlighting the difficulty of balancing robustness with receptiveness.
  • Future Directions: The authors suggest that KAD could be used to benchmark "history-management" interventions, helping developers create models that can admit they were pressured into a mistake without losing their factual grounding.

Summary updated

Abstract

When a language model accepts a false correction in a factual discussion, its next answer can depend on how the conversation continues. We introduce Knowledge Answer Divergence (KAD), an event-conditioned evaluation protocol that distinguishes initial false-answer adoption from subsequent persistence, recovery, and abstention. KAD freezes the conversation at the first explicit adoption of a designated false target and branches independent probes from that exact history. This enables matched comparisons of post-adoption behavior and history-management interventions that a single susceptibility score cannot capture. We evaluate eight local checkpoint conditions and six hosted endpoints, alongside an audited 300-question rerun. In corrected Llama 3.2 Instruct 3B controls, a neutral-history branch yields 42/46 correct responses, compared with 16/46 under full history. A matched cue-placement control changes false adoption by 27.3 percentage points in Qwen 3.5 4B. Twelve independently authored pressure templates demonstrate substantial sensitivity to wording. Together, these findings demonstrate the utility of KAD for evaluating factual reliability beyond initial susceptibility, enabling systematic comparisons of false-answer persistence, recovery conditions and conversational interventions.
BibTeX
@misc{dutta2026knowledge,
  author = {Sujato Dutta and Prithviraj Patil and Chandrajit Bajaj},
  title = {{Knowledge After Collapse: Context and Wording Shape Factual Accessibility in Language Models}},
  year = {2026},
  eprint = {2611.17763},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.17763},
}
Figure