Orthogonal Gradient Descent for Continual Learning

Mehrdad Farajtabar, Navid Azizan, Alex Mott, Ang Li

Abstract

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequential manner, however, neural networks suffer from "catastrophic forgetting"—the tendency to overwrite previously learned information when trained on new data. This paper introduces Orthogonal Gradient Descent (OGD), a method that leverages the overparameterization of modern networks to find update directions that do not interfere with past knowledge.

Introduction

Artificial neural networks excel at learning specific tasks but struggle with continual learning. When a standard network is trained on Task B after Task A, the weight updates usually overwrite the features learned for Task A, a phenomenon known as catastrophic forgetting.

Existing solutions often involve "rehearsal" (storing and re-training on old data, which raises privacy and memory concerns) or "regularization" (penalizing changes to important weights, which can be overly restrictive). The authors propose that the high capacity of overparameterized networks can be used to find "safe" directions for learning that satisfy new task requirements without disrupting previous outputs.

Methods

The core innovation of Orthogonal Gradient Descent (OGD) is the projection of gradients onto a subspace orthogonal to the gradients of past task outputs. The process follows four key steps:

  1. Identify Important Directions: After finishing a task, the model calculates the gradients of its outputs (logits) with respect to its parameters for a small subset of the training data. These represent the directions in parameter space that change the model's predictions the most.
  2. Build an Orthogonal Basis: Using the Gram-Schmidt process, the model maintains a set of orthogonal vectors (a basis) that span the space of these "sensitive" directions from all previous tasks.
  3. Project New Gradients: When training on a new task, the standard loss gradient ($g$) is projected onto the space perpendicular to the basis of previous tasks. This resulting gradient ($\tilde{g}$) is, by definition, orthogonal to the directions that would change previous task outputs.
  4. Update: The model weights are updated using $\tilde{g}$. Because the update is orthogonal to the previous tasks' gradient space, the model's performance on those tasks remains (locally) unchanged.
Conceptual diagram of Orthogonal Gradient Descent (OGD) projecting the gradient g onto a subspace orthogonal to previous task sensitivities to find g-tilde.

A key distinction is that OGD uses the gradient of the model outputs (logits) rather than the gradient of the loss. The loss gradient can be zero for well-learned examples, hiding useful information, whereas the logit gradients always point to how the model "understands" the data.

Experimental Results

The method was evaluated on three standard continual learning benchmarks based on the MNIST dataset: Permuted MNIST, Rotated MNIST, and Split MNIST. OGD consistently matched or outperformed popular baselines like EWC (Elastic Weight Consolidation) and A-GEM (Averaged Gradient Episodic Memory).

Comparison of task accuracies over training steps. OGD (blue) maintains high accuracy on previous tasks as new ones are introduced, unlike standard SGD or EWC.

On the challenging Split MNIST benchmark, OGD achieved near-perfect retention. After learning five consecutive tasks, OGD maintained ~98-99% accuracy across all of them, significantly outperforming A-GEM and EWC, which saw performance drops as more tasks were added. Furthermore, OGD is privacy-friendly; unlike rehearsal methods, it only needs to store a set of gradient vectors rather than original raw images.

Discussion and Limitations

While OGD provides a robust mathematical framework for preventing forgetting, it faces some practical challenges. The size of the orthogonal basis grows with the number of tasks, which may require future compression techniques like Principal Component Analysis (PCA). Additionally, the method's reliance on local linearity means it can be sensitive to high learning rates that move weights beyond the local neighborhood. Finally, extending OGD to work with adaptive optimizers like Adam remains an area for future research.

Summary updated

Abstract

Neural networks are achieving state of the art and sometimes super-human performance on learning tasks across a variety of domains. Whenever these problems require learning in a continual or sequen...
BibTeX
@inproceedings{farajtabar2020orthogonal,
  author = {Mehrdad Farajtabar and Navid Azizan and Alex Mott and Ang Li},
  title = {{Orthogonal Gradient Descent for Continual Learning}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2020},
  url = {https://openreview.net/forum?id=77-DBBRShO8},
}
Figure