TL;DR
This paper introduces FaMe (Factored Mean training), a novel regularization technique for deep neural networks that decomposes weight matrices into smaller factors. By applying multiplicative noise to these factors during training and averaging them at test time, FaMe achieves regularization comparable to Dropout while potentially using fewer parameters and offering a more accurate geometric mean prediction.
What Problem Does This Paper Address?
Training large neural networks, while powerful, is prone to overfitting. These models can easily memorize noise in the training data, leading to poor generalization on unseen data. Regularization techniques are crucial to combat this. Dropout, a highly successful technique, works by randomly dropping units during training, effectively training an ensemble of sub-networks. However, the authors observe that rectified linear unit (ReLU) activations, common in deep networks, can lead to sparse activations, diminishing the effectiveness of Dropout's multiplicative noise. They aim to develop a regularization method that is effective even with sparse activations and potentially more parameter-efficient.
Methods
The core of FaMe lies in weight factorization. Instead of learning a single weight matrix $W$ for a layer, FaMe learns two smaller matrices, $U$ and $V$, such that $W = VU$. During training, multiplicative noise is applied to the output of the first factor ($U$) before it's multiplied by the second factor ($V$). This is conceptually similar to gated models where weights are modulated by a context, but here the modulation is achieved through random noise applied to the factored weights.
At test time, FaMe averages the predictions from these noisy sub-networks. This averaging is achieved by replacing the noisy factors with their expected values, effectively reconstructing the original weight matrix $W = VU$. The authors show that this process approximates the geometric mean of the predictions from the various sub-networks trained.
Key aspects of FaMe:
- Weight Factorization: Decomposes a weight matrix $W$ into $VU$.
- Noise Application: Multiplicative noise is applied after the input is projected onto $U$.
- Ensemble Training: Similar to Dropout, it implicitly trains an ensemble of models.
- Geometric Mean Prediction: At test time, it aims to provide an accurate geometric mean of the ensemble.
- Parameter Efficiency: By restricting the rank of the factors (i.e., the inner dimension of $U$ and $V$), FaMe can potentially use fewer parameters than a standard network.
Datasets
The paper evaluates FaMe on two standard image classification benchmarks:
- MNIST: A dataset of handwritten digits.
- CIFAR-10 and CIFAR-100: Datasets of color images from 10 and 100 classes, respectively.
Key Insights and Ideas
- Dropout's Sparsity Issue: The paper highlights a potential weakness of Dropout when used with ReLU activations, where sparse activations can render the multiplicative noise ineffective. FaMe aims to circumvent this by applying noise to factored weights.
- FaMe as a Gated Model Variant: FaMe can be viewed as a special case of gated models, where the "context" for modulating weights is provided by random noise applied to the factored representation.
- Geometric Mean Approximation: A significant insight is that FaMe's testing procedure provides a better estimate of the true geometric mean of the trained subnetworks compared to Dropout, which, while performing well, doesn't as accurately capture this mean.
- Implicit L2 Regularization: Both FaMe and Dropout exhibit behavior akin to L2 weight decay, with FaMe showing a more pronounced effect. This suggests that the factorization and noise application implicitly regularize the weights.
- Parameter Reduction Potential: By controlling the rank of the factored matrices, FaMe offers a mechanism to significantly reduce the number of parameters while maintaining competitive performance.
Key Results
- MNIST Performance: FaMe achieved a test error of 0.91% (±0.02) on MNIST, outperforming Dropout and Maxout, even when using models with an order of magnitude fewer parameters. For instance, a FaMe model with 3 hidden layers and 512 units per layer had ~93K effective parameters, compared to ~2.3 million for Maxout and ~1.9 million for Gaussian Dropout.
- CIFAR Performance: FaMe was competitive with other recent results on CIFAR-10 and CIFAR-100, achieving a 12.85% error on CIFAR-10. While it didn't outperform Dropout or Maxout in this specific evaluation, the authors attribute this to a smaller hyper-parameter search space and suggest further optimization could improve results.
- Test Cost Decrease: Figure 2 shows that FaMe training allows the test cost to continue decreasing throughout training, indicating effective overfitting prevention, similar to Dropout but with a potentially longer period of improvement.
Limitations and Future Work
- Hyper-parameter Optimization: The authors acknowledge that the hyper-parameter search space for CIFAR was smaller than for MNIST, suggesting that more extensive optimization could lead to better performance.
- Arithmetic vs. Geometric Mean: While FaMe accurately estimates the geometric mean, the paper notes that the arithmetic mean prediction of FaMe subnetworks slightly outperformed the geometric mean prediction, which warrants further investigation.
- Further Exploration of Gated Models: The connection to gated models and the potential for more sophisticated weight modulation strategies could be a fruitful area for future research.
- Rank Restriction Benefits: The paper suggests that restricting the rank of the factor loadings is a way to control free parameters, implying that further research into optimal rank selection could be beneficial.
Summary updated