Learning Neural Networks with Adaptive Regularization

Han Zhao, Yao-Hung Hubert Tsai, Ruslan Salakhutdinov, Geoffrey J. Gordon

Executive Summary

Overfitting remains a major hurdle when training large neural networks on small datasets. This paper introduces AdaReg, an adaptive regularization method that utilizes empirical Bayes to learn correlations between neurons. By allowing neurons to "borrow statistical strength" from one another, AdaReg significantly improves model generalization and optimization compared to traditional static regularization techniques.

The Challenge of Small Datasets

Deep learning's success is heavily reliant on the availability of massive datasets. When data is scarce, large models tend to overfit, memorizing noise rather than learning generalizable patterns. While standard techniques like weight decay ($L_2$ regularization) or Dropout help mitigate this, they are inherently "static"—they do not adapt to the specific geometry of the data or the internal correlations that naturally develop between neurons during training.

The authors observe that during backpropagation, the weight updates for neurons in the same layer are naturally correlated. This raises a critical question: Can we design a prior distribution that actively captures these correlations to help the model generalize better?

Methods: Adaptive Regularization (AdaReg)

The authors propose AdaReg, which abandons the standard assumption that neural network weights are independent and identically distributed (i.i.d.).

Comparison of regularization approaches

Instead of relying on fixed hyperparameters, AdaReg employs an empirical Bayes framework. The core components of this approach include:

  1. Matrix-Variate Normal Prior: Instead of a simple Gaussian prior on each individual weight, AdaReg places a matrix-variate normal prior on the entire weight matrix $W$. This prior uses two covariance matrices ($\Sigma_r$ and $\Sigma_c$) to capture correlations between the rows (fan-in) and columns (fan-out) of the weights.
  2. Kronecker Product Structure: The covariance of the weights is modeled as a Kronecker product ($\Sigma_c \otimes \Sigma_r$). This structure is computationally efficient and specifically designed to capture how neurons "share" information through the learning process.
  3. Empirical Bayes Framework: Rather than picking fixed hyperparameters for the prior, AdaReg estimates them directly from the data. It treats the precision matrices (the inverses of the covariance matrices) as parameters to be optimized alongside the network.
  4. Efficient Optimization: To solve this, the authors developed a Block Coordinate Descent (BCD) algorithm. It alternates between updating the neural network weights using standard backpropagation and updating the precision matrices using a closed-form analytical solution involving Singular Value Decomposition (SVD) and hard thresholding.

Key Insights

  • Borrowing Statistical Strength: By learning the correlations between weights, a neuron that has seen less "informative" data can effectively learn from the experience of other neurons in the same layer. This artificially increases the "effective sample size" during training.
  • Volume Minimization: Mathematically, the AdaReg regularizer acts as an upper bound on the "volume" spanned by the weight vectors. Minimizing this volume forces the model to find a simpler, more correlated representation, which is less prone to overfitting.
  • Spectral Properties: The method encourages the weight matrices to have lower stable ranks and smaller spectral norms. Theoretical deep learning research suggests that these properties are strong indicators of a model's ability to generalize to unseen data.

Key Results

The researchers evaluated AdaReg across several standard benchmarks, varying training set sizes to test its robustness.

Accuracy vs. Train Size on MNIST and CIFAR-10
  • Superior Generalization: On image classification tasks (MNIST and CIFAR-10), AdaReg consistently outperformed weight decay, Dropout, and Batch Normalization. This performance gap was particularly pronounced when the training data was highly limited.
  • Batch Size Independence: Unlike Batch Normalization, which can struggle or behave unpredictably with very large minibatches, AdaReg remains effective and stable across different batch settings.
  • Multitask Success: In the SARCOS robot arm task (a multitask regression dataset involving inverse dynamics), AdaReg achieved the highest "explained variance" across all seven regression tasks compared to other multitask learning baselines.
  • Better Local Optima: Optimization trajectories demonstrated that AdaReg helps the loss function converge to "better" local optima—specifically, those that result in lower loss on the test set, not just the training set.

Limitations and Future Work

While highly effective, AdaReg does introduce additional computational cost. Although the BCD algorithm is efficient ($O(d^3)$), performing SVDs on large weight matrices is inherently more computationally expensive than simple $L_2$ regularization.

Looking forward, the authors suggest that extending this framework to an "online" learning setting (where the model learns from one instance at a time) is a promising direction. Furthermore, they aim to conduct theoretical regret analysis to better understand the method's properties in adaptive optimization scenarios.

Summary updated

Abstract

Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes method. Specifically, we propose to construct a matrix-variate normal prior (on weights) whose covariance matrix has a Kronecker product structure. This structure is designed to capture the correlations in neurons through backpropagation. Under the assumption of this Kronecker factorization, the prior encourages neurons to borrow statistical strength from one another. Hence, it leads to an adaptive and data-dependent regularization when training networks on small datasets. To optimize the model, we present an efficient block coordinate descent algorithm with analytical solutions. Empirically, we demonstrate that the proposed method helps networks converge to local optima with smaller stable ranks and spectral norms. These properties suggest better generalizations and we present empirical results to support this expectation. We also verify the effectiveness of the approach on multiclass classification and multitask regression problems with various network structures. Our code is publicly available at:~\url{https://github.com/yaohungt/Adaptive-Regularization-Neural-Network}.
BibTeX
@inproceedings{zhao2019learning,
  author = {Han Zhao and Yao-Hung Hubert Tsai and Ruslan Salakhutdinov and Geoffrey J. Gordon},
  title = {{Learning Neural Networks with Adaptive Regularization}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2019},
  url = {https://openreview.net/forum?id=O0Kv8Wdked},
}
Figure