Abstract
A common assumption in causal modelling is that the relations between variables are fixed mechanisms. But in reality, these mechanisms often change over time and new data might not fit the original model as well. But is it reasonable to regularly train new models or can we update a single model continually instead? We propose utilizing the field of continual learning to help keep causal models updated over time.
Introduction
In traditional causal modeling, researchers often operate under the assumption that the underlying mechanisms governing a system are static. However, real-world environments are rarely stationary. Consumer preferences shift, economic conditions evolve, and new variables—previously unobserved—frequently enter the system. When these shifts occur, the existing causal model becomes outdated.
This paper proposes a paradigm shift: instead of treating causal discovery and estimation as a one-time process followed by expensive retraining, we should treat it as a continual learning task. By integrating Neural Causal Models (NCMs) with continual learning (CL) techniques, we can update models efficiently, preserve privacy, and maintain previously learned knowledge.
The Challenges of Non-Stationary Environments
The authors identify two primary challenges when updating causal models in response to new data:
- P1 (Change of Probabilities): The causal structure (the graph) remains constant, but the strength or nature of the relationships between variables changes. This results in a distribution shift where the conditional probabilities must be updated.
- P2 (Change of Structure): The fundamental architecture of the system changes. This might involve the introduction of a new variable (e.g., a new regulatory factor) or the addition/removal of a causal link between existing variables.
Standard approaches to these problems usually involve retraining the model from scratch. However, this is often unfeasible. Computational costs can be prohibitive, and more critically, the original training data may no longer be accessible due to storage limits or privacy regulations like GDPR. Furthermore, new data might only provide information about the changed components, making it impossible to reconstruct the entire system from the new data alone.
A Roadmap for Continual Causal Learning
To address these challenges, the authors propose a strategic framework that maps existing Continual Learning techniques onto Neural Causal Models. NCMs are particularly well-suited for this because they parameterize structural assignments using neural networks, which are the primary focus of CL research.
Addressing Distribution Shifts (P1)
To update mechanisms without "catastrophic forgetting"—where the model loses its ability to handle old scenarios while learning new ones—the authors suggest:
- Rehearsal and Pseudorehearsal: Mixing new data with a small subset of stored old data or synthetically generated data from the previous model version.
- Elastic Weight Consolidation (EWC): A regularization technique that penalizes changes to weights that were important for previous tasks, effectively "freezing" the parts of the neural network that represent stable causal mechanisms.
Addressing Structural Changes (P2)
When the graph itself changes, the model requires more than just weight updates; it may require more capacity. The authors suggest:
- Progressive Neural Networks: This involves dynamically expanding the network architecture. When a new variable is introduced, new neurons or layers are added to handle the new causal path, while the weights of the existing, unchanged mechanisms remain frozen or are reused as features.
The Synergy of Causal Modularity
A core insight of this proposal is that causal models are inherently modular. The Independent Mechanisms (IM) assumption states that the mechanism generating one variable is independent of the mechanisms generating others.
This modularity is a natural fit for continual learning. In a modular NCM, if only one relationship in the system changes, we theoretically only need to update the specific neural module corresponding to that relationship. This localized updating prevents the "global" forgetting often seen in standard deep learning models and allows for highly efficient updates.
Future Directions and Meta-Modeling
While this framework is currently conceptual, it opens several avenues for future research. One promising area is the use of knowledge distillation to transfer insights from an old causal graph to a new one, selectively discarding outdated information.
Additionally, the authors propose the development of a "meta-model." This higher-level model would analyze the history of updates to a causal system to predict future shifts. By understanding the trajectory of how causal mechanisms change, we could potentially move from reactive model updates to proactive causal forecasting.
Conclusion
Merging continual learning with neural causal models addresses the practical realities of data science: environments change, and data is often ephemeral. By leveraging causal modularity and CL techniques like EWC and progressive architectures, we can build causal systems that are as dynamic as the worlds they aim to model.
Summary updated