Abstract
Feed-forward neural networks can be understood as a combination of an intermediate representation and a linear hypothesis. While most previous works aim to diversify the representations, we explore the complementary direction by performing an adaptive and data-dependent regularization motivated by the empirical Bayes method. Specifically, we propose to construct a matrix-variate normal prior (on weights) whose covariance matrix has a Kronecker product structure. This structure is designed to capture the correlations in neurons through backpropagation. Under the assumption of this Kronecker factorization, the prior encourages neurons to borrow statistical strength from one another. Hence, it leads to an adaptive and data-dependent regularization when training networks on small datasets. To optimize the model, we present an efficient block coordinate descent algorithm with analytical solutions. Empirically, we demonstrate that the proposed method helps networks converge to local optima with smaller stable ranks and spectral norms. These properties suggest better generalizations and we present empirical results to support this expectation. We also verify the effectiveness of the approach on multiclass classification and multitask regression problems with various network structures.
Introduction
Deep neural networks (DNNs) are notoriously data-hungry. When trained on small datasets, these high-capacity models tend to overfit, memorizing noise rather than learning generalizable patterns. While standard techniques like weight decay ($L_2$ regularization) or Dropout help, they treat weights as largely independent or use fixed, data-independent penalties.
This research identifies a missed opportunity: during backpropagation, the gradients for neurons in the same layer are often highly correlated. Standard regularization doesn't exploit this structure. The challenge is to create a regularization method that is adaptive and data-dependent, allowing the model to learn the relationship between its own internal components to improve performance when data is scarce.
Methodology: The AdaReg Framework
The proposed framework, AdaReg, is built on two main pillars:
- Empirical Bayes Framework: Unlike standard Bayesian methods that require a fixed prior, Empirical Bayes estimates the hyperparameters of the prior distribution directly from the data. This makes the regularization "adaptive" to the specific task and dataset.
- Matrix-Variate Normal Prior: Instead of treating the weight matrix $W$ as a flat vector, AdaReg models it using a matrix-variate normal distribution. Crucially, it assumes the covariance has a Kronecker product structure ($\Sigma_c \otimes \Sigma_r$). This structure separately captures the correlations between the "fan-in" (input) and "fan-out" (output) of neurons.
To make this computationally feasible, the authors developed an efficient Block Coordinate Descent (BCD) algorithm. While optimizing neural networks is usually complex, the authors derived closed-form analytical solutions for the covariance matrices using Singular Value Decomposition (SVD) and a thresholding function. This ensures the method scales well even for layers with thousands of neurons.
Empirical Evaluation
The researchers evaluated AdaReg across several standard benchmarks to test its versatility:
- MNIST & CIFAR-10: Used for multiclass classification. To test the "small data" hypothesis, they varied the training set size (e.g., from 60 to 60,000 samples for MNIST).
- SARCOS: A multitask regression dataset involving the inverse dynamics of a seven-degree-of-freedom robot arm, testing how models share information across related tasks.
Key Results
As shown in the performance curves below, AdaReg consistently outperformed standard baselines like Weight Decay, Dropout, and Batch Normalization, particularly in low-data regimes.
- Superior Generalization: On both MNIST and CIFAR-10, AdaReg provided significant gains when the training set was small. In the SARCOS robotics task, it achieved the highest "explained variance" across all seven regression tasks compared to other multitask learning (MTL) baselines.
- Lower Stable Rank: Empirically, AdaReg leads to solutions with significantly smaller spectral norms and stable ranks. In learning theory, these properties are strongly linked to better generalization bounds.
- Meaningful Correlations: Visualizing the learned covariance matrices showed that the model naturally discovered relationships—for example, in MNIST, the weights for the digits "1" and "7" (which look similar) showed strong positive correlation.
Discussion and Insights
Borrowing Statistical Strength
By learning the correlations between weights, a neuron that has seen very few relevant examples can "borrow" information from a correlated neuron that has. This effectively increases the "sample size" each neuron sees during training. Geometrically, the regularization objective is equivalent to minimizing the "volume" of the weight matrix, forcing the model to find simpler, more robust solutions.
The Softmax Sweet Spot
Ablation studies revealed that applying AdaReg to the final softmax layer yields the best results. While the final layer should exploit correlations between tasks or classes, internal feature layers often benefit more from being decorrelated to maintain diverse representations.
Limitations and Future Work
While effective, applying AdaReg to every layer can sometimes be counterproductive, potentially leading to over-correlated features. Determining exactly which layers benefit most remains an open question. Additionally, while the BCD algorithm is efficient, the SVD step could become a bottleneck for extremely wide layers in massive models. Future research may explore extending AdaReg to online learning settings where data arrives sequentially.
Summary updated