The Limits of Traditional Weight Decay
In deep learning, regularization is essential to prevent models from simply memorizing training data. The industry standard for this is weight decay, which applies a constant penalty to all parameters to keep them small and prevent overfitting.
However, weight decay is fundamentally a "one-size-fits-all" tool. Modern neural networks are highly heterogeneous; a weight matrix in an early convolutional layer might need more freedom to grow than one in a final dense layer. Applying a uniform penalty can be overly restrictive for some parameters—stifling important feature learning—while remaining insufficient for others. A more surgical, adaptive approach is needed to account for the different roles parameters play during training.
Introducing Constrained Parameter Regularization (CPR)
To address the limitations of uniform weight decay, researchers introduce Constrained Parameter Regularization (CPR). Instead of adding a static penalty term to the loss function, CPR treats regularization as a constrained optimization problem. Specifically, it enforces an upper bound ($\kappa$) on a statistical measure, such as the $L_2$-norm, of individual parameter matrices.
To solve this, the authors adapt the augmented Lagrangian method, introducing a Lagrange multiplier ($\lambda$) for each parameter group. This creates a dynamic regularization effect: if a weight matrix's norm exceeds the bound $\kappa$, $\lambda$ increases, applying more "pressure" to shrink those weights. If the weights remain within the bound, the pressure eases. This allows for a more natural path to optimal weight magnitudes compared to the constant pressure of standard weight decay.
Setting the Bound ($\kappa$)
A key challenge in this approach is determining the appropriate bound $\kappa$. The authors propose four distinct strategies:
- Kappa-K: A fixed bound applied uniformly to all matrices.
- Kappa-kI0: A bound proportional to the initial norm of the weights.
- Kappa-WS (Warm Start): The bound is set to the norm reached after a few thousand initial training steps.
- Kappa-IP (Inflection Point): A hyperparameter-free strategy that automatically sets the bound at the first point where the rate of weight growth begins to slow down.
Key Insights and Empirical Results
The authors validated CPR across a diverse set of benchmarks, including computer vision (CIFAR100, ImageNet), language modeling (OpenWebText, PubMedQA), and medical image segmentation (BTCV, BraTS 2020).
Efficiency Gains in Pre-training
One of the most striking findings is CPR's ability to accelerate convergence. When pre-training language models, AdamCPR can reach the same perplexity as a standard AdamW model while using significantly fewer optimization steps. In some cases, CPR achieves peak performance using only two-thirds of the standard training budget.
Robustness to Learning Rates and Fine-Tuning
CPR appears markedly less sensitive to the choice of learning rate compared to weight decay. During the fine-tuning of Large Language Models (LLMs) like Mistral 7B on biomedical data, CPR consistently outperformed the best-tuned AdamW baseline across a wide range of learning rate settings.
Furthermore, CPR helps mitigate catastrophic forgetting. During LLM fine-tuning, models regularized with CPR maintained better performance on general benchmarks while adapting to new domain-specific data, suggesting it preserves pre-trained knowledge more effectively than standard methods.
Broad Applicability
Beyond language models, CPR demonstrated strong performance in vision tasks. When training DeiT on ImageNet, AdamCPR (using Kappa-WS) improved Top-1 accuracy by ~0.6% for small models and ~0.4% for base models over optimized AdamW. It also improved Dice scores across multiple medical segmentation benchmarks when paired with an SGD optimizer, proving its versatility across different optimizer families.
Limitations and Future Work
The primary limitation of CPR is a minor runtime overhead. Because the method requires per-matrix norm calculations and updates to Lagrange multipliers, it can increase training time by roughly 0.4% to 5.8%, depending on model size and hardware. While this overhead is negligible for smaller models or larger batch sizes, it becomes more noticeable as models scale to billions of parameters.
Future work aims to optimize the implementation with custom CUDA kernels to eliminate this computational overhead, paving the way for CPR to be seamlessly integrated into the training pipelines of massive, state-of-the-art foundational models.
Summary updated