Can you make models believe false facts in the prompt?

Ananya Sutradhar, Fabien RogerarXiv 2026 · arXiv

TL;DR

Researchers have developed prompt-conditioned Synthetic Document Finetuning (SDF), a method that allows a single LLM to internalize and act upon arbitrary false facts provided in a prompt tag. By training on thousands of synthetic documents, the model learns to treat these "injected" facts as its own deep-seated knowledge, with belief depth scaling log-linearly with the number of training facts.

What Problem Does This Paper Address?

In AI alignment research, we often need to study "model organisms"—models that hold specific, sometimes counterfactual, beliefs. For example, to test if a model will "play along" during an evaluation, researchers might want to make the model believe it is not currently being tested.

The gold standard for implanting these beliefs is Synthetic Document Finetuning (SDF), where a model is trained on a corpus of documents (news articles, textbooks) written as if the false fact were true. However, standard SDF is computationally expensive because it requires a dedicated fine-tuning run for every single fact you want to inject. This paper tackles the challenge of amortization: Can we train one model to believe any fact we put in its prompt, treating it not as a temporary instruction, but as a fundamental truth?

Methods

The authors introduce prompt-conditioned SDF. Instead of training on one fact, they train a single model on a massive bank of ~10,000 diverse false facts.

  1. Conditioning: Each training document is preceded by a <facts> tag containing the target false fact. The model learns that information inside this tag should be treated as the "ground truth" for the following text.
  2. Training Mixture: To prevent the model from becoming a "yes-man" or losing its general capabilities, the training set includes "anchor" data: true facts in tags, empty tags, and standard instructions where no tag is present.
  3. Reinforcement Learning (RL): The authors used Group Relative Policy Optimization (GRPO) to further refine behavior. The goal was to penalize the model if it "broke character" by citing the prompt (e.g., saying "According to the provided facts...") rather than asserting the fact as its own knowledge.
  4. Jacobian Lens: To see if the model actually believed the lies, they used a "Jacobian lens"—an interpretability tool that decodes internal activations into vocabulary space. This allowed them to check if "falsity-related" tokens (like wrong or incorrect) were surfacing internally while the model was generating "true" sounding answers.
Figure 1: SDF produces one model trained with multiple documents about a single false fact. In prompt-conditioned SDF we train a model with multiple documents for different false facts, but conditioned on the presence of the fact in the <facts> tag. At evaluation-time, we give the prompt-conditioned SDF model a false fact in the <facts> tag of the system prompt and otherwise use it like a regular chat model. We find that even on held-out false facts we never trained on, this approach sometimes makes the prompt-conditioned SDF model act similarly to the SDF model trained on this false fact.

Datasets

  • Synthetic Fact Bank: 10,280 false facts generated by Claude 3.5 Sonnet across 31 domains (e.g., biology, history, memes).
  • Fact Categories: Facts were categorized by "difficulty":
  • Egregious: Contradicts strong world knowledge (e.g., "Light travels slower than sound").
  • Subtle: Contradicts expert knowledge.
  • BKC/AKC: Facts set before or after the model's knowledge cutoff.
  • Evaluation Sets: 25-fact and 100-fact held-out sets, including human-written claims from FEVER and counterfactual edits from RippleEdits.

Key Insights and Ideas

  • Log-Linear Scaling: The most significant finding is that the "belief rate" on facts the model has never seen before scales log-linearly with the number of distinct facts used during training. This suggests that "learning to believe the prompt" is a generalizable skill that improves with data volume.
  • Internal Sincerity: The Jacobian lens results were striking. When the original model sees a false fact in a prompt, its internal activations spike with "falsity" terms. In the prompt-conditioned model, these internal "skepticism" signals vanish, suggesting the model is not just role-playing, but has integrated the fact into its internal representation of reality.
  • Reality Preservation: Interestingly, prompt-conditioned SDF is "safer" than standard SDF. Standard SDF can "break" a model's sense of reality, making it struggle to distinguish real entities from fictional ones. The prompt-conditioned approach maintains a much sharper "reality separation" for facts not currently in the tag.

Key Results

  • Belief Rate: On simple belief questions, the rate rose from 0.09 (original model) to 0.86 (model trained on 10k facts).
  • Generalization: The model successfully applied false facts to complex downstream tasks, such as code debugging or Fermi estimates, often inventing supporting details to justify the false premise.
  • RL Impact: Adding RL significantly reduced the frequency of the model citing the prompt as the source of its information, bringing it nearly level with per-fact SDF on several benchmarks.
Figure 3: Two responses from a prompt-conditioned SDF model given one false fact in the <facts> tag, on two task types: a downstream code debugging task and a causal-inference advice question. Blue marks claims the model asserts as established knowledge of its own and does not mention the fact comes from the prompt. The response also includes confabulated supporting authority and a dismissal of the true fact as outdated information.

Limitations and Future Work

  • Adversarial Weakness: While the model is good at "believing" a fact in a vacuum, it is still more susceptible to "push-back" (user contradiction) than a model specifically fine-tuned on a single fact.
  • Model Scale: The study focused on models in the 7B to 27B range. Larger models have stronger "priors" (pre-existing knowledge), which might make them harder to "brainwash" using this method.
  • The "Liar's Paradox": The authors warn that using RL to stop a model from citing the prompt might just be teaching it to be a better liar (hiding the source of its belief) rather than actually deepening the belief itself. Future work is needed to disentangle "acting as if" from "believing."

Summary updated

Abstract

Making LLMs believe false facts is useful for alignment evaluations and for building model organisms. Previous work explored doing so with Synthetic Document Finetuning (SDF). But SDF requires fine-tuning one new model for each set of injected false beliefs. To enable larger scale SDF-powered alignment evaluations, we attempt to produce a single model that believes any fact placed in a <facts> tag in the prompt. To do so, we explore doing prompt-conditioned SDF: fine-tuning a model on synthetic documents covering thousands of facts, with each document conditioned on a fact presented in the prompt. We evaluate how deeply the model believes the facts provided in the prompt using evaluations from prior work and a new Jacobian lens evaluation. We find that the belief depth on unseen facts scales roughly log-linearly with the number of training facts, making this a cheap, promising avenue for making models believe false facts, though preliminary experiments on applying it to alignment audits on the small model we studied showed negative results due to these small models often already acting as if they believed facts provided in regular honeypot system prompts. Using RL to reduce the rate at which models verbalize following a false fact in the prompt, on top of prompt-conditioned SDF, generalizes to held-out evals and almost matches SDF on some evaluations. But training against one of our metrics for belief depth makes it harder to trust those evaluations.
BibTeX
@misc{sutradhar2026can,
  author = {Ananya Sutradhar and Fabien Roger},
  title = {{Can you make models believe false facts in the prompt?}},
  year = {2026},
  eprint = {2611.23938},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23938},
}
Figure