When a neural network learns a sequence of tasks, it often suffers from catastrophic forgetting—the tendency to overwrite past knowledge when adapting to new data. For years, machine learning research treated forgetting primarily as a function of time, network capacity, or optimization algorithms. However, a deeper look at how networks update their internal representations reveals a more nuanced driver: the semantic and statistical relationships between the tasks themselves.
The way sequential tasks relate to one another dictates how gradients flow through a network, determining whether new knowledge builds upon old foundations or aggressively dismantles them.
The Two Faces of Task Similarity
Intuitively, one might assume that learning similar tasks sequentially is always beneficial, while learning completely unrelated tasks causes the most disruption. The reality is a double-edged sword. Task relationship impacts forgetting through two distinct failure modes: representation interference (when tasks are too distinct) and gradient confusion (when tasks are too similar).
1. When Dissimilarity Hurts: Representation Interference
When a model transitions between semantically distinct tasks, it faces a structural alignment problem. In Exploring the Challenges towards Lifelong Fact Learning, Mohamed Elhoseiny et al. demonstrated that when sequential tasks lack conceptual overlap, models experience significantly higher rates of forgetting. Without shared semantic anchors, the network cannot reuse existing features and is forced to drastically alter its weight configurations to accommodate the new domain.
This phenomenon is highly sensitive to the network's training regime. As explored by Jacopo Graldi et al. in The Importance of Being Lazy: Scaling Limits of Continual Learning, distinct tasks pose a severe risk of representation interference unless the model is pushed into a "lazy" training regime where internal features remain relatively stable. When a model actively learns "rich" features across highly divergent tasks, the updates required for the new task inevitably distort the representations established for the old one. This pattern holds true across modalities; for instance, in speech recognition, Piotr Despot-Mładanowicz et al. in Balancing Specialization and Generalization Trade-Off for Speech Recognition Models observed that knowledge retention is inherently higher when a specialized downstream task shares strong similarities with the original training data.
2. When Similarity Hurts: Gradient Confusion
Conversely, high task similarity introduces a more insidious problem. When two tasks are highly related or share a structured format, they utilize the exact same sub-networks and parameter pathways. If the tasks contain subtle differences, the gradients generated by the new task will target and aggressively overwrite the precise parameters holding the older knowledge.
High Task Dissimilarity:
Task A Pathway: [==]----->
Task B Pathway: -----> [==] (Representation Interference if capacity is low)
High Task Similarity:
Task A Pathway: [==]----->
Task B Pathway: [==]-----> (Gradient Confusion: Direct parameter overwriting)
This counterintuitive effect is prominent in structured information extraction and language modeling. In Curriculum-Meta Learning for Order-Robust Continual Relation Extraction, Tongtong Wu et al. found that semantic similarity is a primary driver of forgetting because overlapping contexts lead to severe model confusion. When a model processes sequential tasks within the same semantic neighborhood, the boundaries between old and new concepts blur.
In Continual Learning of Long Topic Sequences in Neural Information Retrieval, Laure Soulier et al. articulated this as a complex trade-off: while high similarity can facilitate positive knowledge transfer, the subtle differences between related topics trigger aggressive gradient updates that erase prior nuances.
This is not just an artifact of weight-tuning in neural networks; it even manifests in parameter-free contexts. Analyzing Large Language Models (LLMs) performing in-context continual learning, Guangyu Li et al. in Understanding Generalization and Forgetting in In-Context Continual Learning proved that statistically similar tasks cause greater attention-based interference than completely irrelevant ones. Processing entirely unrelated tasks can actually preserve prior task performance more effectively because the attention mechanisms do not conflate the historical context.
Similarly, when fine-tuning LLMs on sequential facts, Howard Chen et al. in Continual Memorization of Factoids in Large Language Models discovered that forgetting is most pronounced when the model is trained on identical structures, such as sequential blocks of factoid triplets. The structural similarity forces the updates into the same localized parameter regions, erasing previously memorized facts.
Architectural Interventions: The Role of Activation Functions
Because the relationship between tasks dictates how gradients interact with parameters, standard optimization techniques often fall short. Early foundational work by Ian J. Goodfellow et al. in An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks revealed that the performance ranking of different activation functions shifts dramatically depending on whether the sequential tasks are similar or dissimilar. This task-dependent behavior means that mitigating forgetting requires architectural components that can dynamically manage representation overlap.
If the root cause of forgetting is that tasks step on each other's parameter footprints, the solution is to enforce sparsity—ensuring that different tasks activate distinct, isolated pathways within the network.
Standard Activation (Dense Overlap):
Task A: █ █ █ █ █
Task B: █ █ █ █ █ <-- High Overwrite Risk
Sparse Activation (Isolated Pathways):
Task A: █ ░ █ ░ ░
Task B: ░ █ ░ █ ░ <-- Low Overwrite Risk
To embed this robustness directly into the network structure, Qingfeng Lan et al. introduced "elephant activation functions" in Elephant Neural Networks: Born to Be a Continual Learner. By designing activation functions that inherently produce both sparse representations and sparse gradients, the authors created a system where updates for a new task are naturally routed away from parameters utilized by prior tasks.
When activation functions enforce sparsity, the network reduces its reliance on the shifting semantics of the task stream. Whether two sequential tasks are subtly similar or wildly divergent, the architectural constraint prevents their gradients from colliding in the same parameter space, offering a structural solution to a fundamentally relational problem.
Key Foundational Work
An Empirical Investigation of Catastrophic Forgeting in Gradient-Based Neural Networks (2014) — contributed 7 concept elements to this direction.
Exploring the Challenges towards Lifelong Fact Learning (2018) — contributed 1 concept elements to this direction.