In the study of continual learning, intuition suggests a linear relationship: the more a new task differs from what a model already knows, the more it should struggle to retain its old knowledge. However, a growing body of research reveals a more complex, non-monotonic reality. Neural networks—and even human learners—experience a "danger zone" of intermediate similarity where catastrophic forgetting is most severe.
The Danger Zone of Intermediate Similarity
Standard intuition assumes that forgetting scales with task divergence. Yet, work by Sebastian Lee, Sebastian Goldt, and Andrew M Saxe in Continual Learning in the Teacher-Student Setup: Impact of Task Similarity demonstrates that forgetting actually peaks when tasks are moderately similar.
This creates a U-shaped interference pattern:
- High Similarity: If a new task is nearly identical to the old one, the model can reuse existing representations with minimal weight updates, preserving memory.
- Low Similarity (Orthogonality): If tasks are completely unrelated, the model can utilize its high-dimensional "null space" to learn the new task without overwriting the weights used for the old one.
- Intermediate Similarity: In the middle, tasks are similar enough to "tempt" the network into using the same internal features, but different enough that the required weight updates for the new task actively corrupt the logic of the old one.
Maslow’s Hammer: Re-use vs. Activation
To explain why this happens, Sebastian Lee and colleagues proposed the Maslow’s Hammer hypothesis in Maslow's Hammer for Catastrophic Forgetting: Node Re-Use vs Node Activation. The name refers to the proverb "if all you have is a hammer, everything looks like a nail."
When a network encounters a new task, it faces a fundamental trade-off between node re-use (leveraging old knowledge for fast transfer) and node activation (recruiting fresh, unused capacity to protect memory). Intermediate tasks are the "nails" that trigger the hammer: they are similar enough that gradient descent tries to adapt existing specialized nodes rather than starting from scratch. This hybrid approach leads to an optimization bottleneck where the network struggles to balance conflicting gradients, resulting in the highest levels of representational erasure.
Disentangling Features and Readouts
The severity of this interference depends heavily on where the similarity lies. Naoki Hiratani argues in Disentangling and mitigating the impact of task similarity for continual learning that we must distinguish between input feature similarity and output readout similarity.
The most "catastrophic" scenario occurs when tasks share high input feature similarity but require low (or conflicting) output readouts. In this case, the network attempts to use the same "detectors" for two different purposes. For example, if a model learns to recognize a "circle" as a "zero" in one task and then must learn to recognize it as a "ball" in the next, the gradient update for "ball" will aggressively overwrite the "zero" association because they share the same underlying feature representation.
The Buffer of Overparameterization
Does making models larger solve this? Not necessarily. Daniel Goldfarb and Daniel Soudry show in The Joint Effect of Task Similarity and Overparameterization on Catastrophic Forgetting that while overparameterization provides a buffer by allowing updates to reside in a "null space," this benefit is least effective at that same intermediate similarity level.
Furthermore, Jacopo Graldi and Lorenzo Noci demonstrate that scaling only helps if it pushes the network into a "lazy" regime where features barely change. If the model continues to learn "rich" features—which is usually desirable for performance—wider networks forget just as much as small ones when faced with similar tasks.
A Shared Computational Burden
This interference isn't just a quirk of artificial neural networks; it appears to be a fundamental property of learning systems. Eleanor Holton and Lukas Braun found that humans and ANNs show nearly identical patterns of transfer and interference in Humans and neural networks show similar patterns of transfer and interference during continual learning.
Similarly, Nicholas Menghi and Christian F. Doeller observed in The effects of task similarity during representation learning in brains and neural networks that the brain must actively "orthogonalize"—or decorrelate—its internal representations to overcome the cognitive interference triggered by structural similarity.
Toward Optimal Protocols
Understanding this non-monotonicity allows for better training strategies. Francesco Mori and colleagues used statistical physics in Optimal Protocols for Continual Learning via Statistical Physics and Control Theory to derive a "focus then revise" strategy. Instead of uniform practice, they suggest that by carefully selecting the order and weight of tasks, one can leverage intermediate similarity to facilitate specialization rather than destruction, turning a potential bottleneck into a tool for more efficient learning.