Surprise Alone Learns Lies: Evidence-Gated Continual Learning for LLMs from Streaming Claims

Abdelkader Metales, Miriam GrabenheinricharXiv 2026 · arXiv

TL;DR

Current methods for teaching LLMs new information on the fly often rely on "surprise" to prioritize what to learn. This paper demonstrates that surprise-driven learning is a "poison magnet": because lies contradict what a model knows, they are highly surprising and thus absorbed at alarming rates. The authors propose an evidence-gated framework using multiple checks—including the model's own prior beliefs and external verifiers—to filter out falsehoods while still allowing for legitimate updates.

What Problem Does This Paper Address?

As we deploy LLMs in dynamic environments (like a university assistant tracking changing regulations), they need to undergo continual learning—updating their knowledge without full retraining. A common heuristic for this is "learning by surprise": the model prioritizes data that has a high negative log-likelihood (NLL), assuming it represents novel information.

However, the authors identify a critical security flaw: surprise is not truth. If an attacker tells a model that "Paris is the capital of Germany," that claim is highly surprising to a model that knows the truth. Under a surprise-only regime, the model doesn't just see a lie; it sees a "high-priority learning opportunity." This paper explores how to defend against such "poisoned" data streams without accidentally blocking legitimate new information.

Methods

The researchers tested several "keys" or gates that a claim must pass before being written into the model's weights via LoRA (Low-Rank Adaptation).

  1. Surprise Key: Is this information new? (Standard baseline).
  2. Prior-Conflict Key: Does this contradict what the original, frozen base model believes? This acts as a "cheap" internal filter that doesn't require an external judge.
  3. Verifier Key: An "LLM-as-a-judge" (e.g., Gemini 1.5 Flash or Phi-4) that evaluates the claim, sometimes with access to a trusted source.
  4. Statistical/Regression Test: After a trial training step, does the model's performance on unrelated, previously known facts degrade? If so, the update is rolled back.

The full proposed system, shown in Figure 1, combines these gates to decide whether to learn, reject, or hold a claim for further review.

Figure 1: The acceptance decision for one claim (sura/runner.py). A claim is learned only if it is new, uncontradicted and verified. Claims are processed in chunks of 16; each key can be switched off for the ablations.

Datasets

The study uses a controlled benchmark consisting of three types of claims:

  • New Facts: Information about 120 "invented people" (e.g., birthplace, profession) that the model couldn't possibly know.
  • Known-Fact Lies: Counterfactual rewrites of real-world facts from the CounterFact dataset (e.g., changing a person's actual birthplace).
  • Legitimate Updates: Real facts that have changed, requiring the model to overwrite its prior knowledge.

Key Insights and Ideas

  • Surprise is a Poison Magnet: Surprise-driven learners are incredibly vulnerable. They absorbed nearly 100% of lies about facts they already knew. As shown in Figure 2, surprise-only systems (blue line) consistently assert the highest share of lies across all poison rates.
  • The Prior is a Free Filter: A model’s own internal confidence can flag contradictions. By checking the "gap" between what the model wants to say and what the claim says, the system can filter out 76% of known-fact lies without calling an expensive external judge.
  • The "Evidence Route" Paradox: To handle legitimate updates (e.g., a new CEO), you must allow the model to overwrite its prior. The authors found that routing contradicted claims to a "trusted source" check is the only way to recover these updates safely. However, if an attacker can "plant" fake sources, the system becomes more vulnerable than using a judge alone.
Figure 2: Share of all streamed lies asserted after learning, by poison rate (oracle verifier; Qwen2.5-1.5B; 6 systems × 3 poison rates × 3 seeds; results/main).

Key Results

The quantitative results highlight the trade-offs between learning speed and safety. As seen in Table 3 (Figure 5), the Verifier Key achieved the best balance, keeping 98% of old facts while learning 86% of new ones.

  • Absorption Rates: Surprise-only models absorbed 84–93% of all streamed lies.
  • Judge Efficiency: Using the Prior-Conflict key as a triage filter saved 43% of calls to the more expensive LLM judge.
  • Real-World Judges: When using real LLMs (like Phi-4) as judges, they were found to be "strict but safe"—they rarely accepted lies but often rejected true information if it lacked a clear source.
Table 3: Learning and retention at 30% poison (oracle verifier). Means over 3 seeds; seed standard deviations are at most 0.05, except for the verifier key with the uncalibrated test (0.18 at 30%, 0.35 at 50% poison), whose rollbacks made learning unstable.

Limitations and Future Work

The authors acknowledge several hurdles for real-world deployment:

  • Adaptive Attackers: If an attacker has access to the model, they can craft lies that "sit close" to the model's own beliefs, bypassing the Prior-Conflict key.
  • Scale: The study focused on smaller models (1.5B to 3B parameters) and short, atomic factual claims. Whether these dynamics hold for 70B+ models or complex, multi-hop reasoning remains to be seen.
  • Source Trust: The "Evidence Route" relies entirely on the assumption that sources cannot be forged—a difficult guarantee in the age of AI-generated misinformation.

Future work suggests moving toward corroboration: requiring multiple independent "chains of transmission" before a model commits a surprising new fact to its long-term weights.

Summary updated

Abstract

Continual learning methods for large language models increasingly decide what to learn by surprise, keeping the inputs the model finds least predictable. This rule is unsafe when the stream contains lies. A planted falsehood about something the model knows has low probability under the model's belief, so it is highly surprising by construction, and a surprise-driven learner absorbs nearly every such lie, at a rate similar to random selection with the same budget. We study the problem with LoRA-adapted students (Qwen2.5-1.5B, Qwen2.5-3B and Llama-3.2-3B) that learn from a stream of true and false claims about invented people and real CounterFact facts. We report four findings. (1) A verifier key, a judge that must approve each claim, removes lies to the extent that its verifier is accurate; real LLM judges are cautious and reject unsourced truths and even source-backed updates. (2) Calibrated regression tests on unrelated knowledge slow learning down without filtering out lies. (3) The original model's own prior flags claims that contradict what it knows, without a judge. It separates lies about known facts from true new facts (AUC 0.78-0.83 across three models) but not from legitimate updates of the same kind of fact (AUC 0.53). It saves 43% of judge calls, but an attacker with access to the model can choose lies that pass it. (4) If trusted sources exist that an attacker cannot plant, routing contradicted claims to evidence recovers legitimate updates and the combined system has the lowest poison rate among learned claims; when an attacker plants matching sources for 30% of the lies, the system ends up worse than the judge alone. The acceptance signals answer different questions, and none of them decides on its own whether a claim is true.
BibTeX
@misc{metales2026surprise,
  author = {Abdelkader Metales and Miriam Grabenheinrich},
  title = {{Surprise Alone Learns Lies: Evidence-Gated Continual Learning for LLMs from Streaming Claims}},
  year = {2026},
  eprint = {2610.39840},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.39840},
}
Figure