TL;DR
This paper re-examines Elastic Weight Consolidation (EWC), a method designed to prevent neural networks from forgetting previously learned tasks when trained sequentially. The authors argue that the standard EWC formulation, which uses multiple quadratic penalties for each past task, is theoretically inconsistent and can lead to "double-counting" data from earlier tasks. They propose a more theoretically sound, single-penalty EWC that anchors to the most recently learned parameters, offering a simpler and potentially more robust approach.
What Problem Does This Paper Address?
Neural networks are powerful tools, but they often struggle with sequential learning. When a network trained on one task is then trained on a new task, it tends to "forget" what it learned before, a phenomenon called catastrophic forgetting. Elastic Weight Consolidation (EWC) was introduced as a way to mitigate this by penalizing changes to network weights that were important for previous tasks. However, the original EWC formulation, especially when extended to more than two tasks, uses multiple penalties, each anchored to the parameters learned for a specific past task. This paper questions the theoretical justification for these multiple, separate penalties.
Methods
The authors revisit the derivation of EWC, drawing connections to Laplace approximation and Bayesian inference. They start by re-deriving the two-task EWC case, showing its equivalence to a diagonalized Laplace approximation. The core of their contribution lies in extending this to more than two tasks. They argue that a recursive application of Laplace approximation naturally leads to a single quadratic penalty, anchored at the parameters learned for the most recent task. This is because the penalty for the most recent task implicitly incorporates the importance of weights from all preceding tasks. They contrast this with the original EWC's approach of using multiple penalties, each centered on the parameters from a specific earlier task. The paper also proposes a multi-penalty version with "debiased" penalty centers as an alternative for scenarios requiring revisiting past datasets.
Datasets
Not discussed in the paper.
Key Insights and Ideas
- Inconsistency in Multi-Task EWC: The paper argues that the standard EWC approach of using separate quadratic penalties for each past task is theoretically flawed. This can lead to "double-counting" the importance of certain weights, as a penalty anchored to a later task's parameters already implicitly accounts for the importance of weights from earlier tasks.
- The Power of the Latest Anchor: A more theoretically sound approach for multi-task learning, derived from recursive Laplace approximation, suggests that only a single quadratic penalty is needed. This penalty should be anchored to the parameters learned for the most recent task. This single penalty implicitly captures the information from all previous tasks.
- Simplicity and Efficiency: The proposed single-penalty EWC is not only more theoretically consistent but also simpler and more efficient, as it avoids maintaining multiple penalty terms and their associated parameters. This also leads to constant storage costs regardless of the number of tasks.
- Revisiting Data: The paper acknowledges that in some cases, it might be desirable to revisit past datasets. For this, they propose a modified multi-penalty EWC with adjusted penalty centers, which can be used to refine approximations.
Key Results
The paper presents a theoretical derivation and argument rather than empirical results. The key "result" is the theoretical justification for a single-penalty EWC over the multi-penalty version for sequential learning. This theoretical insight suggests that the single-penalty approach is more consistent with Bayesian principles and avoids potential biases introduced by multiple penalties.
Limitations and Future Work
The authors acknowledge that the original EWC formulation might be used in practice and that their multi-penalty version with debiased centers could be useful for specific applications requiring dataset revisiting. They suggest that their derived single-penalty EWC should be sufficient for most cases and offers constant storage costs. The paper focuses on the theoretical underpinnings and does not present empirical evaluations of the proposed single-penalty EWC against the original multi-penalty EWC or other methods. Future work could involve empirical validation of the proposed single-penalty EWC and its performance benefits.
Summary updated