TL;DR
Constrained Parameter Regularization (CPR) replaces traditional uniform weight decay by enforcing individual upper bounds on the norm of each parameter matrix. Solved dynamically using an adapted augmented Lagrangian method, CPR eliminates the need for uniform penalty tuning and reduces the training budget for large language models by up to 33% while boosting downstream performance.
What Problem Does This Paper Address?
To prevent deep neural networks from overfitting, researchers typically rely on weight decay ($L_2$ regularization). However, standard weight decay applies a single, static penalty coefficient uniformly across every single parameter matrix in a network.
This "one-size-fits-all" approach is fundamentally limited. Different layers capture different features and operate at varying scales; a penalty that is perfectly tuned for one layer might be overly restrictive for another, stifling feature learning, or too weak elsewhere, allowing overfitting. While scheduling weight decay throughout training can help, it introduces complex, task-sensitive hyperparameters that are notoriously tedious to tune.
Methods
The authors introduce Constrained Parameter Regularization (CPR), reframing regularization as a constrained optimization problem rather than a soft penalty addition. Instead of constantly pulling weights toward zero, CPR enforces an explicit upper bound ($\kappa^j$) on a statistical measure (like the squared $L_2$-norm) of each individual parameter matrix $\theta^j$.
To solve this, they adapt the classic augmented Lagrangian method to deep learning:
- Each parameter matrix gets its own constraint and its own Lagrange multiplier ($\lambda^j$).
- During training, if a matrix's norm stays below its allocated bound $\kappa^j$, the constraint is inactive ($\lambda^j = 0$), and the optimizer updates the weights without any regularization interference.
- If the norm violates the upper bound, $\lambda^j$ dynamically increases, applying localized regularization pressure to pull the weights back within bounds.
The authors propose two primary strategies to initialize these upper bounds ($\kappa^j$):
- Kappa-WS (Warm Start): The model trains normally for a set number of steps ($s$), and $\kappa^j$ is initialized to whatever norm the matrix naturally achieved at that step.
- Kappa-IP (Inflection Point): A completely hyperparameter-free strategy. It automatically sets the bound at the first inflection point of the regularization function during early training—the exact moment the rate of weight growth begins to naturally saturate.
Datasets
CPR was rigorously evaluated across multiple domains and architectures using the following datasets:
- Image Classification: CIFAR-100, CIFAR-100-C (for noise robustness), and ImageNet.
- Language Modeling: OpenWebText (for pre-training GPT-2 small and medium models) and PubMedQA (for fine-tuning Mistral-7B using LoRA).
- Medical Image Segmentation: Multi-Atlas Labeling Beyond the Cranial Vault (BTCV), Medical Segmentation Decathlon (Heart), and the Brain Tumor Segmentation challenge (BraTS 2020).
Key Insights and Ideas
- Embracing Heterogeneity: CPR allows the network to naturally allocate regularization pressure where it is actually needed over time, accommodating the distinct roles of different layers.
- Altered Training Dynamics: Weight decay causes a rapid initial drop in training loss but can lead to early stagnation. CPR allows weights to grow freely early on, leading to a more linear, stable, and ultimately superior optimization path.
- Self-Initializing Regularization: The success of the Kappa-IP strategy reveals that a network's early training dynamics contain all the information required to self-initialize optimal regularization bounds, rendering manual weight decay tuning obsolete for pre-training.
Key Results
- 33% Compute Savings in LLMs: During GPT-2 pre-training, AdamCPR achieved the same validation perplexity as a fully tuned AdamW baseline while utilizing only 2/3 of the optimization steps, significantly cutting down wall-clock training time.
- Consistent Language Model Improvements: Across both GPT-2 small and medium architectures, CPR consistently outperformed AdamW across various learning rates.
- Vision Transformer Breakthroughs: On ImageNet pre-training with DeiT-Base, AdamCPR with the hyperparameter-free Kappa-IP initialization achieved a Top-1 accuracy of 80.95%, outperforming the standard AdamW baseline (80.56%).
- Mitigating Catastrophic Forgetting: When fine-tuning Mistral-7B on PubMedQA, CPR not only improved downstream accuracy on the target task but also significantly reduced performance degradation on unrelated benchmarks like TruthfulQA.
Limitations and Future Work
- Runtime Overhead: Because CPR computes norms and updates multipliers independently for each matrix, it introduces a minor computational overhead. While negligible for smaller models, it caused up to a 6% increase in runtime per step for a 1.1B parameter model. The authors note that a dedicated CUDA-optimized
for_eachimplementation could resolve this. - Diminishing Returns: The benefits of CPR fade in regimes where regularization has a minimal impact overall, such as training underparameterized models on massive datasets where overfitting isn't a primary bottleneck.
- Future Directions: The authors suggest scaling CPR to massive LLMs (10B+ parameters) and exploring alternative statistical constraints beyond the standard $L_2$-norm.
Summary updated