Abstract
Current machine learning algorithms are successful in learning clearly defined tasks from large i.i.d. data. Continual learning (CL) requires learning without iid-ness and developing algorithms capable of knowledge retention and transfer, the latter can be boosted through systematic generalization. Dropping the i.i.d. assumption requires replacing it with another hypothesis. While there are several candidates, here we advocate that the independent mechanism assumption (IM) (Schölkopf et al. 2012) is a useful hypothesis for representing knowledge in a form, that makes it easy to adapt to new tasks in CL. Specifically, we review several types of distribution shifts that are common in CL and point out in which way a system that represents knowledge in form of causal modules may outperform monolithic counterparts in CL. Intuitively, the efficacy of IM solution emerges since: (i) causal modules learn mechanisms invariant across domains; (ii) if causal mechanisms must be updated, modularity can enable efficient and sparse updates.
Introduction
Traditional machine learning relies heavily on the IID assumption, which posits that training and test data are independent and identically distributed. Continual Learning (CL) fundamentally challenges this premise, as agents encounter a sequence of tasks where data distributions change over time. In standard "monolithic" neural networks—architectures where all parameters are potentially involved in every computation—these distribution shifts often lead to catastrophic forgetting. When the model learns new information, it frequently overwrites the weights necessary for previous tasks. While existing solutions like Elastic Weight Consolidation (EWC) attempt to mitigate this by regularizing weight changes, they often face a trade-off between plasticity (learning new things) and stability (retaining old things).
The Independent Mechanisms Assumption
This research proposes that the solution lies in how we structure knowledge. By adopting the Independent Mechanisms (IM) assumption from causal inference, we can represent the world as a collection of autonomous, modular mechanisms. Under this assumption, the mechanism governing one part of a system (e.g., a mathematical operation) does not inform or interfere with another (e.g., the input distribution). This modularity allows for sparse updates: when a specific part of the environment changes, only the corresponding module needs to be updated, leaving the rest of the system's knowledge intact.
Methodology: Modular Continual Learners (MCL)
The authors introduce Modular Continual Learners (MCL), which utilize several key architectural components to enforce modularity:
- Mixture of Experts (MoE): The system consists of $M$ independent modules that compete to process observations.
- Key-Value Attention: A routing mechanism determines which module is best suited for a given task description ($T$).
- Functional Bottlenecks: Inspired by Neural Production Systems, these limit the information flow to each module, forcing them to learn specific, sparse mechanisms rather than entangled representations.
Experimental Setup
The framework was evaluated using a synthetic math equations domain where the model must learn operations (e.g., +, -, *) on numbers sampled from $[-1, 1]$. Two primary scenarios were tested:
- Stream 1 (Standard Task Shifts): New operations are introduced sequentially.
- Stream 2 (Hidden Shifts): The meaning of an existing operation changes abruptly (e.g., the "+" symbol shifts from representing $x_1 + x_2$ to $(x_1 + x_2)/5$).
Key Results and Discussion
The MCL architecture was compared against EWC and monolithic baselines. The results highlight a critical advantage for modular systems in dynamic environments.
In standard task shifts, MCL eventually matches the performance of EWC as it learns to route tasks correctly. However, the true strength of the modular approach is evident in Stream 2 (Hidden Shifts). As seen in the performance metrics, MCL significantly outperforms EWC when the underlying meaning of a task changes. EWC's strategy of protecting old weights prevents it from adapting to the new "shifted" logic. In contrast, MCL's modularity allows it to update only the relevant module or routing logic, effectively re-learning the specific mechanism without a system-wide collapse.
Conclusion
By moving from monolithic architectures to those grounded in the Independent Mechanisms assumption, we can create AI systems that are more robust to the distribution shifts inherent in Continual Learning. While MCL demonstrates the power of modularity and sparse updates, significant challenges remain in automatically discovering these modules from raw, high-dimensional data. Future research should focus on identifying the inductive biases that allow models to naturally decompose complex problems into independent causal mechanisms.
Summary updated