Improving Deep Learning Optimization through Constrained Parameter Regularization

Jörg K. H. Franke, Michael Hefenbrock, Gregor Koehler, Frank Hutter

TL;DR

Constrained Parameter Regularization (CPR) replaces traditional uniform weight decay by enforcing individual upper bounds on the norm of each parameter matrix. Solved dynamically using an adapted augmented Lagrangian method, CPR eliminates the need for uniform penalty tuning and reduces the training budget for large language models by up to 33% while boosting downstream performance.

What Problem Does This Paper Address?

To prevent deep neural networks from overfitting, researchers typically rely on weight decay ($L_2$ regularization). However, standard weight decay applies a single, static penalty coefficient uniformly across every single parameter matrix in a network.

This "one-size-fits-all" approach is fundamentally limited. Different layers capture different features and operate at varying scales; a penalty that is perfectly tuned for one layer might be overly restrictive for another, stifling feature learning, or too weak elsewhere, allowing overfitting. While scheduling weight decay throughout training can help, it introduces complex, task-sensitive hyperparameters that are notoriously tedious to tune.

Methods

The authors introduce Constrained Parameter Regularization (CPR), reframing regularization as a constrained optimization problem rather than a soft penalty addition. Instead of constantly pulling weights toward zero, CPR enforces an explicit upper bound ($\kappa^j$) on a statistical measure (like the squared $L_2$-norm) of each individual parameter matrix $\theta^j$.

To solve this, they adapt the classic augmented Lagrangian method to deep learning:

  1. Each parameter matrix gets its own constraint and its own Lagrange multiplier ($\lambda^j$).
  2. During training, if a matrix's norm stays below its allocated bound $\kappa^j$, the constraint is inactive ($\lambda^j = 0$), and the optimizer updates the weights without any regularization interference.
  3. If the norm violates the upper bound, $\lambda^j$ dynamically increases, applying localized regularization pressure to pull the weights back within bounds.

The authors propose two primary strategies to initialize these upper bounds ($\kappa^j$):

  • Kappa-WS (Warm Start): The model trains normally for a set number of steps ($s$), and $\kappa^j$ is initialized to whatever norm the matrix naturally achieved at that step.
  • Kappa-IP (Inflection Point): A completely hyperparameter-free strategy. It automatically sets the bound at the first inflection point of the regularization function during early training—the exact moment the rate of weight growth begins to naturally saturate.

Datasets

CPR was rigorously evaluated across multiple domains and architectures using the following datasets:

  • Image Classification: CIFAR-100, CIFAR-100-C (for noise robustness), and ImageNet.
  • Language Modeling: OpenWebText (for pre-training GPT-2 small and medium models) and PubMedQA (for fine-tuning Mistral-7B using LoRA).
  • Medical Image Segmentation: Multi-Atlas Labeling Beyond the Cranial Vault (BTCV), Medical Segmentation Decathlon (Heart), and the Brain Tumor Segmentation challenge (BraTS 2020).

Key Insights and Ideas

  • Embracing Heterogeneity: CPR allows the network to naturally allocate regularization pressure where it is actually needed over time, accommodating the distinct roles of different layers.
  • Altered Training Dynamics: Weight decay causes a rapid initial drop in training loss but can lead to early stagnation. CPR allows weights to grow freely early on, leading to a more linear, stable, and ultimately superior optimization path.
  • Self-Initializing Regularization: The success of the Kappa-IP strategy reveals that a network's early training dynamics contain all the information required to self-initialize optimal regularization bounds, rendering manual weight decay tuning obsolete for pre-training.

Key Results

  • 33% Compute Savings in LLMs: During GPT-2 pre-training, AdamCPR achieved the same validation perplexity as a fully tuned AdamW baseline while utilizing only 2/3 of the optimization steps, significantly cutting down wall-clock training time.
Figure 1: GPT2s training using Adam with weight decay or CPR (Kappa-IP). AdamCPR outper- forms AdamW with the same budget and only re- quires 2/3 of the budget to reach the same score.
  • Consistent Language Model Improvements: Across both GPT-2 small and medium architectures, CPR consistently outperformed AdamW across various learning rates.
Figure 3: Perplexity (↓) ± std across three random seeds of GPT2s and GPT2m trained on OpenWeb- Text with AdamW (left) and AdamCPR with Kappa-IP (middle) and AdamCPR with Kappa-WS (right). We use a learning rate warm-up of 5k steps. The CPR with the hyperparameter-free strategy Kappa-IP outperforms weight decay but also CPR with warm start.
  • Vision Transformer Breakthroughs: On ImageNet pre-training with DeiT-Base, AdamCPR with the hyperparameter-free Kappa-IP initialization achieved a Top-1 accuracy of 80.95%, outperforming the standard AdamW baseline (80.56%).
  • Mitigating Catastrophic Forgetting: When fine-tuning Mistral-7B on PubMedQA, CPR not only improved downstream accuracy on the target task but also significantly reduced performance degradation on unrelated benchmarks like TruthfulQA.

Limitations and Future Work

  • Runtime Overhead: Because CPR computes norms and updates multipliers independently for each matrix, it introduces a minor computational overhead. While negligible for smaller models, it caused up to a 6% increase in runtime per step for a 1.1B parameter model. The authors note that a dedicated CUDA-optimized for_each implementation could resolve this.
  • Diminishing Returns: The benefits of CPR fade in regimes where regularization has a minimal impact overall, such as training underparameterized models on massive datasets where overfitting isn't a primary bottleneck.
  • Future Directions: The authors suggest scaling CPR to massive LLMs (10B+ parameters) and exploring alternative statistical constraints beyond the standard $L_2$-norm.

Summary updated

Abstract

Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.
BibTeX
@inproceedings{franke2024improving,
  author = {Jörg K. H. Franke and Michael Hefenbrock and Gregor Koehler and Frank Hutter},
  title = {{Improving Deep Learning Optimization through Constrained Parameter Regularization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=rCXTkIhkbF},
}
Figure