When a neural network learns a sequence of tasks, it typically suffers from catastrophic forgetting: optimizing parameters for a new task inadvertently destroys the performance on previously learned ones. Traditionally, researchers have fought this by freezing important weights, isolating parameters, or replaying old data.
However, a powerful geometric perspective reframes the problem entirely. Forgetting is not an inevitable consequence of parameter updates; rather, it is a symptom of the curvature of the loss landscape. By actively guiding models toward flat minima—broad, shallow valleys in the error surface rather than steep, narrow ravines—we can create a parameter buffer that inherently tolerates subsequent updates.
The Geometry of Forgetting
To understand why flat minima prevent forgetting, visualize the loss landscape of a neural network as a terrain where the height represents the error rate.
Loss
^
| /\ Sharp Minimum (Fragile)
| / \ /\ Moving weights slightly causes
| / \_/ \ a massive spike in loss.
| / \
| / \____ Flat Minimum (Robust)
| / \____ Weights can shift freely
| / \_ without increasing loss.
+---------------------------------------------------> Parameters
When a model trains on Task A, standard gradient descent seeks any parameter configuration that minimizes the loss. If it settles into a sharp minimum, the loss increases rapidly with even minor changes to the weights. When the model subsequently adapts to Task B, the parameters must shift. Even a tiny displacement pushes the model up the steep walls of Task A's loss ravine, causing performance to plummet.
As Mirzadeh et al. demonstrate, the magnitude of forgetting is fundamentally bounded by two factors: the total displacement of the weights and the local curvature (the eigenvalues of the Hessian matrix) at the minimum. If we minimize the curvature—effectively widening the valley—the loss increases slowly around the minimum. This geometric property acts as a safety buffer: the parameter adjustments required to learn new information result in minimal performance degradation for previously acquired tasks (Mirzadeh et al.).
This insight shifts the focus from reactive post-hoc corrections to proactive landscape engineering. As Shi et al. and Shi et al. argue, optimizing for neighborhood flatness early on prepares a resilient weight space that accommodates future task integration without requiring complex correction strategies.
Explicit Optimization: Sharpness-Aware Minimization
The most direct way to find these robust regions is through explicit optimization constraints, primarily derived from Sharpness-Aware Minimization (SAM). Instead of minimizing loss at a single point, SAM-based approaches minimize the maximum loss within a small neighborhood around the current parameters.
- Worst-Case Perturbations: Deng et al. leverage this in their FS-DGPM framework, explicitly smoothing the loss landscape by minimizing the loss at a worst-case weight shift. This ensures that subsequent updates do not cause abrupt performance drops.
- Stabilized Ascent: Standard SAM can be unstable in lifelong learning. To address this, Mordido et al. introduced Lookbehind-SAM, which uses multi-step ascent to find more challenging worst-case perturbations while maintaining stability via weight interpolation, achieving a superior trade-off between low loss and low sharpness.
- Pre-training Synergies: Mehta et al. discovered that while pre-trained models naturally settle into flatter minima than randomly initialized ones, explicitly optimizing for neighborhood flatness using SAM during sequential fine-tuning further amplifies their resistance to forgetting.
Implicit Flattening: Noise, Regularization, and Optimizers
Beyond explicit geometric constraints, certain training regimes, architectural choices, and optimization algorithms implicitly bias models toward flatter regions of the loss landscape.
Noise Injection and Regularization
Deliberately introducing uncertainty during training forces the model to find solutions that are robust to parameter variations. Lee demonstrates that injecting noise into both the input data and the network weights (double perturbation) effectively flattens the landscape, mitigating forgetting in task-free continual learning. Similarly, Le Hai et al. show that treating dropout rates as learnable, task-specific variables (Continual Variational Dropout) steers optimization toward flat minima by inducing data-dependent regularization.
Unsupervised Objectives and Self-Supervision
The choice of loss function itself alters the landscape geometry. The authors of Representational Continuity for Unsupervised Continual Learning found that training with unsupervised objectives promotes the formation of a flatter and smoother loss landscape compared to supervised training, enhancing robustness to parameter updates. In a similar vein, Sun et al. integrate self-supervised learning objectives during new task acquisition to promote a flatter parameter space, which stabilizes subsequent weight averaging for long-term knowledge retention.
Zeroth-Order Optimization
Perhaps the most surprising source of implicit flattening comes from derivative-free optimization. Feng et al. (ZeroFlow) and Bian et al. (C-Flat) demonstrate that zeroth-order (ZO) optimization naturally tends to bypass sharp minima and converge toward flatter regions compared to standard backpropagation. Because ZO optimization estimates gradients by sampling function values in a local neighborhood, it inherently favors wide, shallow basins where the average neighborhood loss is low.
Beyond Continual Learning: A Unifying Geometric Principle
The benefits of loss landscape flattening extend far beyond traditional continual learning. Any machine learning paradigm that involves parameter shifts, model combinations, or vulnerability to subsequent fine-tuning can be improved by targeting flat minima.
Machine Unlearning
Machine unlearning aims to erase specific hazardous or private data from a trained model. However, standard unlearning often leaves the model in a narrow, fragile minimum where a malicious actor can easily "re-teach" the forgotten information using a fraction of the original data.
To build resilience against such relearning attacks, Wang et al. introduce Invariant LLM Unlearning (ILU), which minimizes the squared gradient norm of a proxy loss to force the model into a flat region where the unlearned state acts as a stationary point. This makes it mathematically difficult for subsequent fine-tuning to shift parameters back toward the original knowledge. Similarly, Fan et al. and Yan et al. (PRISM) show that enforcing dual-space smoothness ensures the unlearned state remains permanent and resilient to adversarial recovery.
Model Merging and Multi-Task Coordination
When combining multiple task-specific models into a single network without retraining, "parameter interference" typically degrades performance. Lee et al. resolve this via Sharpness-Aware Fine-Tuning (SAFT). Because model merging acts as a parameter perturbation, fine-tuning individual models to reside in flat minima allows them to tolerate the interference of other task vectors without losing their specialized capabilities.
In concurrent multi-task learning, Wu et al. (MT-COOL) extend the concept of flat minima to inter-task stability. By injecting random noise into the features of auxiliary tasks during a primary task's update, they force the model to find regions in the loss landscape that are explicitly robust to task interference, effectively eliminating negative transfer.
| Application | Role of Flat Minima | Key Benefit |
|---|---|---|
| Continual Learning | Provides a geometric buffer for weight updates | Prevents catastrophic forgetting (Mirzadeh et al.) |
| Machine Unlearning | Anchors the unlearned state as a stable stationary point | Resists relearning and fine-tuning attacks (Wang et al.) |
| Model Merging | Maximizes tolerance to parameter perturbations | Reduces parameter interference during blending (Lee et al.) |
| Multi-Task Learning | Insulates task representations from concurrent updates | Eliminates negative transfer between tasks (Wu et al.) |
Key Foundational Work
Understanding the Role of Training Regimes in Continual Learning (2020) — contributed 4 concept elements to this direction.
Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning (2021) — contributed 2 concept elements to this direction.
AlterSGD: Finding Flat Minima for Continual Learning by Alternative Training (2021) — contributed 2 concept elements to this direction.