The Persistence of Pre-trained Knowledge in Large Language Models

This direction is split into other directions: pre-training knowledge resilience. Its last version stays readable and citable here.

Version 1 of 150 papers70 conceptsfrozen 2026-10-04

When we train a Large Language Model (LLM), we typically follow a two-stage pipeline: a massive pre-training phase on raw internet data, followed by a targeted fine-tuning or alignment phase to teach the model specific tasks or safety constraints. For a long time, researchers assumed that fine-tuning could effectively "overwrite" or "mask" undesirable parts of the pre-training data.

However, a growing body of evidence suggests that knowledge acquired during pre-training is fundamentally different from knowledge acquired later. It is not just a matter of scale; pre-trained representations are structurally resilient, creating a "persistence" that makes them nearly impossible to fully excise through standard unlearning or editing techniques.

The Pre-training Transition

To understand why pre-trained knowledge sticks, we have to look at how a model's learning mechanism changes during its lifecycle. Boxi Cao and colleagues found that pre-training serves as a critical transition point. While randomly initialized models suffer from "catastrophic forgetting"—where new data immediately erases the old—pre-trained weights provide a foundation that allows for incremental accumulation.

This stability is a double-edged sword. Once a concept is baked into the base weights, it becomes part of the model's "long-term memory." Samuel Gehman and Suchin Gururangan demonstrated this with toxicity: even when models are "detoxified" through fine-tuning, the underlying toxic associations from the pre-training corpus remain latent in the parameters, ready to resurface under the right conditions.

Why Foundational Knowledge Resists Erasure

The difficulty of removing a piece of information is directly tied to its "depth" in the training history. Several researchers have identified three main reasons why pre-trained knowledge is so stubborn:

  1. The Frequency Moat: Aravind Krishnan and Marius Mosbach found that the more frequently a fact appears in the pre-training data, the more resistant it is to unlearning. Even if you train the model to stop generating the fact, the internal weights still assign it a high probability.
  2. Structural Ossification: Zac Hatfield-Dodds showed that even tiny amounts of repeated data in the early stages can "hijack" the model's circuits (like induction heads), forcing them to memorize sequences rather than learn to generalize. This creates a structural rigidity that later fine-tuning cannot easily undo.
  3. Domain-Dependent Persistence: Not all subjects are equally easy to edit. Yinyi Luo and Yixuan Li observed that certain academic or medical domains are more "deeply embedded" and resistant to modification than others, suggesting that the model's internal knowledge graph is not uniformly plastic.

The Ghost in the Machine

When we attempt to "unlearn" a pre-trained entity—like a famous fictional character or a private individual—we often end up with what Weitao Ma and Xiaocheng Feng call a "ghost." The model might learn to refuse to answer questions about the entity, but the entity's features still exist in the latent space.

This "ghosting" effect means that "unlearning" is often just "hiding." Yingzi Ma and Jiongxiao Wang found that current methods merely mask information behind a refusal response, leaving the model vulnerable to privacy attacks that can still extract the "forgotten" data. Similarly, John Abascal and Alina Oprea showed that even if you fine-tune a model with Differential Privacy, the original pre-training signals still leak through.

The Failure of Alignment

A common hope was that Reinforcement Learning from Human Feedback (RLHF) would act as a "privacy wash." However, Aneesh Pappu and Jamie Hayes have shown that RLHF is largely ineffective at erasing memorization from earlier stages. Verbatim snippets learned during pre-training or Supervised Fine-Tuning (SFT) tend to persist through the entire alignment process.

This suggests that safety and privacy cannot be "bolted on" at the end. As Tomasz Korbak argues, incorporating preferences directly into the pre-training phase is far more effective than trying to fix a model after it has already internalized a massive, uncurated corpus.

Collateral Damage and the Path Forward

The final hurdle in unlearning pre-trained knowledge is the risk of "catastrophic utility loss." Because pre-trained concepts are so foundational, trying to rip them out often damages the model's general reasoning. Kazuki Egashira and Go Irie found that while models can easily forget recently fine-tuned data, attempting to force the removal of pre-trained concepts causes a severe degradation in overall intelligence.

Current research is shifting toward more surgical interventions. Methods like Model State Arithmetic, proposed by Keivan Rezaei, attempt to use training history to "subtract" specific influences. Others, like Zexi Li, are exploring whether sophisticated knowledge editing tools like AlphaEdit can reach the foundational layers that standard unlearning misses.

Ultimately, the field is realizing that a model's "first impressions" during pre-training are its most lasting ones. Understanding the structural difference between these early weights and later updates is the key to building models that can truly forget.

Papers in this version

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_the_persistence_of_pre_trained_knowledge_c2541434,
  title = {The Persistence of Pre-trained Knowledge in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-persistence-of-pre-trained-knowledge-in-large-language-models/rdr_c25414341df1480dac3264312c559a24},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_the_persistence_of_pre_trained_knowledge_c2541434_v1,
  title = {The Persistence of Pre-trained Knowledge in Large Language Models},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/the-persistence-of-pre-trained-knowledge-in-large-language-models/rdr_c25414341df1480dac3264312c559a24/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}