Abstract
Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from the problem of catastrophic forgetting; they forget how to solve previous tasks after being trained on a new task, despite having the essential capacity to solve both tasks if they were trained on both simultaneously. In this paper, we propose to address this issue from a parameter space perspective and study an approach to restrict the direction of the gradient updates to avoid forgetting previously-learned data. We present the Orthogonal Gradient Descent (OGD) method, which accomplishes this goal by projecting the gradients from new tasks onto a subspace in which the neural network output on previous task does not change and the projected gradient is still in a useful direction for learning the new task. Our approach utilizes the high capacity of a neural network more efficiently and does not require storing the previously learned data that might raise privacy concerns. Experiments on common benchmarks reveal the effectiveness of the proposed OGD method.
Introduction
The Achilles' heel of modern neural networks is catastrophic forgetting. When a model is trained on a sequence of tasks (Task A, then Task B, then Task C), the weight updates for Task B often overwrite the features necessary for Task A. This happens because standard optimizers like Stochastic Gradient Descent (SGD) are "oblivious" to the past; they only care about minimizing the loss for the current batch of data. Ideally, an intelligent agent should learn continuously without requiring the storage of all previous data (replay) or the unsustainable growth of the model architecture.
Methodology: Orthogonal Gradient Descent
The core idea of Orthogonal Gradient Descent (OGD) is to move through the model's weight space to learn a new task while ensuring the output for previous tasks remains unchanged. This is achieved through a projection mechanism in the parameter space.
- Gradient Memory: After finishing a task, the model calculates the gradients of its outputs (logits) with respect to its parameters for a subset of the training data. These gradients represent the directions in parameter space that most affect the model's predictions for that task.
- Orthogonal Projection: When training on a new task, OGD takes the standard gradient ($g$) and projects it onto a subspace that is perpendicular (orthogonal) to all previously stored gradients.
- OGD-GTL Variant: To save memory, the authors propose OGD-GTL (Ground Truth Logit), which only stores the gradient for the correct label's logit rather than every possible output class.
- Gram-Schmidt Process: The stored gradients are orthogonalized using the Gram-Schmidt process to create an efficient basis for this "protected" subspace.
By moving only in these orthogonal directions, the model's predictions for previous tasks stay locally constant, effectively shielding old knowledge from being overwritten.
Experimental Setup
The authors evaluated OGD on three standard continual learning benchmarks based on the MNIST digit dataset:
- Permuted MNIST: Each task involves classifying digits where the pixels have been shuffled by a fixed random permutation.
- Rotated MNIST: Each task involves classifying digits rotated by a specific angle (e.g., 0°, 10°, 20°).
- Split MNIST: The 10 digit classes are split into five tasks, each containing two distinct digits (e.g., Task 1 is 0 vs 1, Task 2 is 2 vs 3).
Key Results
OGD demonstrated state-of-the-art performance across the benchmarks, often outperforming popular baselines like EWC (Elastic Weight Consolidation) and A-GEM (Averaged Gradient Episodic Memory).
In the Split MNIST benchmark, OGD achieved near-perfect accuracy across all tasks, performing almost identically to Multi-Task Learning (MTL)—the theoretical upper bound where the model has full access to all data simultaneously. Furthermore, OGD-GTL proved efficient, requiring only a small number of stored gradients (e.g., 200 per task) to prevent forgetting.
Discussion and Future Work
The success of OGD relies on the over-parameterization of modern neural networks. In high-dimensional parameter spaces, there are typically many "free" directions to move that do not interfere with previous constraints. Additionally, using logit gradients rather than loss gradients captures more essential information about the model's function, as loss gradients can vanish for well-learned examples.
Future work may address the linear growth of storage requirements by using compression techniques like PCA. Additionally, while OGD is effective for similar tasks, it faces challenges when tasks are extremely dissimilar (e.g., rotations > 90°), suggesting that the neighborhood of valid parameters for different tasks may not always overlap sufficiently.
Summary updated