Unified Factored Representations Resist Catastrophic Forgetting

Joel Lehman, Kenneth O. StanleyarXiv 2026 · arXiv

TL;DR

This paper proposes the Representational Forgetting Hypothesis (RFH): catastrophic forgetting is not an inherent flaw of neural networks, but a side-effect of "fractured" internal representations. By ensuring networks develop Unified Factored Representations (UFR)—where domain regularities are captured holistically—models can resist forgetting during fine-tuning without any specialized algorithms or architectural changes.

What Problem Does This Paper Address?

Neural networks are notoriously bad at "continual learning." When you fine-tune a model on new data, it often abruptly loses its grasp on previously learned information—a phenomenon known as catastrophic forgetting.

Standard solutions usually involve extrinsic "band-aids": freezing important weights (Elastic Weight Consolidation), replaying old data, or growing the architecture. Joel Lehman and Kenneth Stanley take a different path. They argue that forgetting happens because standard gradient descent (SGD) produces Fractured Entangled Representations (FER). In an FER, features are messy and globally entangled; a small update to change a local feature (like opening a mouth on a face) inadvertently corrupts unrelated features (like the eyes or the overall face shape) because the weights aren't organized to respect the domain's underlying structure.

Methods

The authors introduce the Cyclic Fine-tuning Paradigm to test their hypothesis. The process is simple:

  1. Initial Model: Start with a model that fits a target function across the full domain.
  2. Cyclic Fine-tuning: Repeatedly fine-tune the model on only a subset of the input space, moving it toward a variation of the target and then back to the original.
  3. Validation: Measure how much the model "forgot" on the held-out portion of the domain where no fine-tuning occurred.
Overview of the Cyclic Fine-tuning Paradigm. A model is trained on a full domain, then cycled between variations using only a subset of the data. Forgetting is measured on the held-out portion.

They compare standard SGD-trained networks against UFR networks. Since we don't yet have a general algorithm to "learn" UFR, they source them from two places:

  • Hand-designed networks: For simple 1D math functions, they manually set weights to perfectly capture symmetries.
  • Picbreeder CPPNs: Compositional Pattern Producing Networks evolved through human-in-the-loop selection, which naturally encourages interpretable, holistic representations.

To prove that the representation is the key factor (and not just the specific architecture), they also use Symmetry Loss and Representation Distillation to "teach" UFR-like properties to standard networks during their initial training phase.

Datasets

The study utilizes two primary experimental domains:

  1. 1D Function Regression: Targets include symmetric functions like the absolute value ($y = |x|$) and more complex multi-level versions.
  2. Picbreeder Images: High-resolution images (Skull, Butterfly, Apple) generated by CPPNs. These serve as a "challenge domain" where regularities are complex and qualitative rather than purely mathematical.
Three target functions of increasing complexity used in the 1D regression tasks.

Key Insights and Ideas

  • The "Adjacent Possible": In a UFR network, the local neighborhood of the weights (the "adjacent possible") is rich with semantically meaningful variations. For example, changing a single weight in a UFR "Skull" network might smoothly open its mouth. In an FER network, the same weight change produces incoherent noise.
  • Compatibility is Crucial: UFR only resists forgetting if the new information is compatible with its internal factoring. If you try to fine-tune a symmetric UFR network on an asymmetric task (like painting an 'X' on one side of a face), the UFR breaks, and the model forgets just as fast as any other.
  • Gradations Matter: Forgetting is further reduced when the model is guided through small, incremental steps ("gradations") toward the new target. This suggests that maintaining representational quality requires attending to what a network is "ready" to learn.
Comparison of weight sweeps. In UFR (top), perturbing a weight leads to coherent changes (mouth opening). In FER (bottom), it leads to incoherent artifacts.

Key Results

  • Quantitative Resistance: In the 1D domain, UFR networks showed significantly lower validation loss after 20 cycles compared to SGD-trained models. Distilling UFR properties into standard models closed much of this gap.
  • Qualitative Coherence: In image tasks, UFR networks maintained the global structure of the image (e.g., the face remained a face) even after 30 cycles of fine-tuning on a small region. In contrast, SGD networks progressively accumulated color artifacts and structural degradation.
  • Capacity as a Liability: Interestingly, adding "free capacity" (extra neurons) to a UFR network actually increased forgetting. The authors hypothesize that gradient descent greedily uses this extra space to build new, entangled features that disrupt the original unified representation.
A hand-designed UFR network for the 1D domain that captures symmetry by splitting and recombining inputs.

Limitations and Future Work

The biggest hurdle is that we currently lack a scalable, general-purpose algorithm to discover and enforce UFR in large-scale models like LLMs. While the authors showed that Symmetry Loss and Distillation work, these require knowing the regularities beforehand or having a "teacher" model.

Furthermore, "elaborating" on a UFR—adding new, complex knowledge without breaking the existing elegant factoring—remains an open challenge. Preliminary experiments showed that naive fine-tuning often "breaks" the UFR rather than extending it. Future research into how models can autonomously apprehend and amortize regularities into their weights could be a major step toward truly continual AI.

Summary updated

Abstract

This paper explores an unexpected connection between the quality of a neural network's (NN's) internal representation and its vulnerability to catastrophic forgetting. The representational forgetting hypothesis is that networks with unified, factored representation (UFR) can resist catastrophic forgetting without any modification to the underlying fine-tuning algorithm or network architecture. The key insight is that catastrophic forgetting may largely result from fractured entangled representation (FER): fine-tuning on only part of the input domain results in disruptive changes for other input patterns, because of how individual features are globally entangled. In contrast, if a gradient update is compatible with how a UFR network factors its representation, such an update may result in coordinated change across the input domain, alleviating catastrophic forgetting. Experiments in two domains provide evidence for this hypothesis: (1) an illustrative 1D regression task, in which UFR networks constructed by hand are compared against networks trained from scratch with SGD, and (2) an image-regression task with NNs previously shown to exhibit UFR. In both settings, UFR networks forget substantially less than their FER counterparts when repeatedly fine-tuned on UFR-compatible variations. Further experiments show that a training loss that distills UFR representations into fresh NNs reduces forgetting; along with several additional controls, these results help isolate representation as the critical factor. Together these results suggest that representational quality is an underappreciated contributor to catastrophic forgetting, and that the encouragement and maintenance of such quality may require attending to what patterns a particular NN is ready to learn.
BibTeX
@misc{lehman2026unified,
  author = {Joel Lehman and Kenneth O. Stanley},
  title = {{Unified Factored Representations Resist Catastrophic Forgetting}},
  year = {2026},
  eprint = {2611.11020},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.11020},
}
Figure