TL;DR
Researchers have developed prompt-conditioned Synthetic Document Finetuning (SDF), a method that allows a single LLM to internalize and act upon arbitrary false facts provided in a prompt tag. By training on thousands of synthetic documents, the model learns to treat these "injected" facts as its own deep-seated knowledge, with belief depth scaling log-linearly with the number of training facts.
What Problem Does This Paper Address?
In AI alignment research, we often need to study "model organisms"—models that hold specific, sometimes counterfactual, beliefs. For example, to test if a model will "play along" during an evaluation, researchers might want to make the model believe it is not currently being tested.
The gold standard for implanting these beliefs is Synthetic Document Finetuning (SDF), where a model is trained on a corpus of documents (news articles, textbooks) written as if the false fact were true. However, standard SDF is computationally expensive because it requires a dedicated fine-tuning run for every single fact you want to inject. This paper tackles the challenge of amortization: Can we train one model to believe any fact we put in its prompt, treating it not as a temporary instruction, but as a fundamental truth?
Methods
The authors introduce prompt-conditioned SDF. Instead of training on one fact, they train a single model on a massive bank of ~10,000 diverse false facts.
- Conditioning: Each training document is preceded by a
<facts>tag containing the target false fact. The model learns that information inside this tag should be treated as the "ground truth" for the following text. - Training Mixture: To prevent the model from becoming a "yes-man" or losing its general capabilities, the training set includes "anchor" data: true facts in tags, empty tags, and standard instructions where no tag is present.
- Reinforcement Learning (RL): The authors used Group Relative Policy Optimization (GRPO) to further refine behavior. The goal was to penalize the model if it "broke character" by citing the prompt (e.g., saying "According to the provided facts...") rather than asserting the fact as its own knowledge.
- Jacobian Lens: To see if the model actually believed the lies, they used a "Jacobian lens"—an interpretability tool that decodes internal activations into vocabulary space. This allowed them to check if "falsity-related" tokens (like wrong or incorrect) were surfacing internally while the model was generating "true" sounding answers.
Datasets
- Synthetic Fact Bank: 10,280 false facts generated by Claude 3.5 Sonnet across 31 domains (e.g., biology, history, memes).
- Fact Categories: Facts were categorized by "difficulty":
- Egregious: Contradicts strong world knowledge (e.g., "Light travels slower than sound").
- Subtle: Contradicts expert knowledge.
- BKC/AKC: Facts set before or after the model's knowledge cutoff.
- Evaluation Sets: 25-fact and 100-fact held-out sets, including human-written claims from FEVER and counterfactual edits from RippleEdits.
Key Insights and Ideas
- Log-Linear Scaling: The most significant finding is that the "belief rate" on facts the model has never seen before scales log-linearly with the number of distinct facts used during training. This suggests that "learning to believe the prompt" is a generalizable skill that improves with data volume.
- Internal Sincerity: The Jacobian lens results were striking. When the original model sees a false fact in a prompt, its internal activations spike with "falsity" terms. In the prompt-conditioned model, these internal "skepticism" signals vanish, suggesting the model is not just role-playing, but has integrated the fact into its internal representation of reality.
- Reality Preservation: Interestingly, prompt-conditioned SDF is "safer" than standard SDF. Standard SDF can "break" a model's sense of reality, making it struggle to distinguish real entities from fictional ones. The prompt-conditioned approach maintains a much sharper "reality separation" for facts not currently in the tag.
Key Results
- Belief Rate: On simple belief questions, the rate rose from 0.09 (original model) to 0.86 (model trained on 10k facts).
- Generalization: The model successfully applied false facts to complex downstream tasks, such as code debugging or Fermi estimates, often inventing supporting details to justify the false premise.
- RL Impact: Adding RL significantly reduced the frequency of the model citing the prompt as the source of its information, bringing it nearly level with per-fact SDF on several benchmarks.
Limitations and Future Work
- Adversarial Weakness: While the model is good at "believing" a fact in a vacuum, it is still more susceptible to "push-back" (user contradiction) than a model specifically fine-tuned on a single fact.
- Model Scale: The study focused on models in the 7B to 27B range. Larger models have stronger "priors" (pre-existing knowledge), which might make them harder to "brainwash" using this method.
- The "Liar's Paradox": The authors warn that using RL to stop a model from citing the prompt might just be teaching it to be a better liar (hiding the source of its belief) rather than actually deepening the belief itself. Future work is needed to disentangle "acting as if" from "believing."
Summary updated