Improving Deep Learning Optimization through Constrained Parameter Regularization

Jörg K. H. Franke, Michael Hefenbrock, Gregor Koehler, Frank Hutter

Abstract

Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.

Introduction: Beyond Global Weight Decay

In deep learning, weight decay is the standard tool for preventing overfitting by adding a constant penalty to the loss function based on weight magnitude. However, weight decay is a "one-size-fits-all" approach, applying the same penalty coefficient to every parameter regardless of its layer or function. This is often suboptimal; some layers may require more capacity to capture complex features, while others should be more strictly constrained. While researchers have tried scheduling weight decay, this usually introduces even more hyperparameters to tune. Constrained Parameter Regularization (CPR) addresses this by providing individualized, adaptive regularization without manual tuning.

Methods: Constraint-Based Optimization

CPR shifts from penalty-based regularization to constraint-based optimization. Instead of a penalty term, it enforces a hard constraint: the statistical measure (e.g., $L_2$-norm) of each parameter matrix must stay below an upper bound, $\kappa$. This is solved using an adaptation of the Augmented Lagrangian method, maintaining a separate Lagrange multiplier ($\lambda$) for every parameter matrix. If a weight norm exceeds $\kappa$, $\lambda$ increases to apply pressure and push the weights back down; otherwise, the pressure vanishes.

The authors propose two primary strategies for setting these bounds ($\kappa$):

  • Kappa-WS (Warm Start): The model trains unconstrained for a few thousand steps, after which current weight norms are "frozen" as upper bounds for the remainder of training.
  • Kappa-IP (Inflection Point): A hyperparameter-free version that automatically sets the bound for each layer at the moment weight growth starts to slow down (the inflection point).
Comparison of AdamW and AdamCPR (Kappa-IP and Kappa-WS) performance across learning rates and regularization settings on a fine-tuning task.

Experimental Evaluation

CPR was validated across a diverse array of benchmarks to prove its generality:

  • Computer Vision: CIFAR100 and ImageNet-1k (Vision Transformers).
  • Language Modeling: GPT-2 pre-training on the OpenWebText corpus.
  • Fine-tuning: Mistral 7B fine-tuning using LoRA on biomedical QA data (PubMedQA).
  • Medical Imaging: Segmentation tasks on BTCV, Heart, and BraTS 2020.

Key Results and Insights

Efficiency and Speedup

In GPT-2 pre-training, AdamCPR reached the same perplexity score as a tuned AdamW baseline in only 66% of the optimization steps, demonstrating significant computational savings and faster convergence.

Perplexity vs. Optimization Steps for GPT-2 pre-training, showing AdamCPR reaching target perplexity significantly faster than AdamW.

Performance Gains

  • Vision: On ImageNet, a DeiT-Base model with CPR achieved 81.19% Top-1 accuracy, surpassing the best-tuned AdamW baseline (80.56%).
  • Medical Imaging: CPR improved Dice scores across all medical benchmarks, proving effective with both SGD and Adam optimizers.
  • LoRA Fine-tuning: CPR consistently outperformed AdamW across various learning rates during Mistral 7B fine-tuning, showing better adaptation to specialized data and mitigating catastrophic forgetting on general benchmarks.

Individualized Pressure and Low Overhead

Unlike weight decay, which is always "on," CPR only intervenes when specific layers grow too large, allowing the model to find its own optimal weight distribution. Despite its mathematical complexity, CPR adds negligible runtime overhead (typically <1%, and up to 6% for very large models) because it requires no extra gradient passes.

Limitations and Future Work

While overhead is low, it scales with the number of parameter groups, reaching 5.8% for a 1.1B parameter model. CPR is most effective when weight regularization is a primary bottleneck; its benefits are less pronounced for very small models relative to dataset size. Future work includes exploring other statistical measures (e.g., infinity or spectral norms) and testing on larger-scale distributed systems.

Summary updated

Abstract

Regularization is a critical component in deep learning. The most commonly used approach, weight decay, applies a constant penalty coefficient uniformly across all parameters. This may be overly restrictive for some parameters, while insufficient for others. To address this, we present Constrained Parameter Regularization (CPR) as an alternative to traditional weight decay. Unlike the uniform application of a single penalty, CPR enforces an upper bound on a statistical measure, such as the L2-norm, of individual parameter matrices. Consequently, learning becomes a constraint optimization problem, which we tackle using an adaptation of the augmented Lagrangian method. CPR introduces only a minor runtime overhead and only requires setting an upper bound. We propose simple yet efficient mechanisms for initializing this bound, making CPR rely on no hyperparameter or one, akin to weight decay. Our empirical studies on computer vision and language modeling tasks demonstrate CPR's effectiveness. The results show that CPR can outperform traditional weight decay and increase performance in pre-training and fine-tuning.
BibTeX
@inproceedings{franke2024improving,
  author = {Jörg K. H. Franke and Michael Hefenbrock and Gregor Koehler and Frank Hutter},
  title = {{Improving Deep Learning Optimization through Constrained Parameter Regularization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2024},
  url = {https://openreview.net/forum?id=rCXTkIhkbF},
}
Figure