Abstract: Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
Introduction
Dropout is one of the most successful tools in the deep learning toolbox for preventing overfitting, yet our theoretical understanding of why it works remains incomplete. While often viewed as a form of ensemble learning or noise injection, this research bridges the gap between deep learning heuristics and classical signal processing to identify the specific mathematical structure Dropout forces upon a network's weights.
The authors reframe Dropout through the lens of Frame Theory and Information Theory, specifically examining Denoising Autoencoders (DAEs) where Dropout is applied to the encoder. They pose a fundamental question: if we want to recover a signal perfectly after some of its components are randomly deleted (erased), what is the optimal shape for the encoding matrix?
Dropout as Signal Recovery
The paper draws a direct parallel between Dropout and the "analog coding" problem—recovering a signal from erasures in a communication channel. By treating the weight matrix of a layer as a "frame" (an overcomplete basis), applying Dropout becomes mathematically equivalent to randomly removing columns from this frame.
In signal processing, the optimal structure to survive random erasures is an Equiangular Tight Frame (ETF). In an ETF, all vectors (filters) have the same length, and the angle between any two vectors is identical and as large as possible, reaching what is known as the "Welch bound." This represents the "most independent" a set of vectors can be when there are more vectors than dimensions.
Coherence Regularization
Because reaching a perfect ETF is a difficult non-convex optimization problem, the authors propose a "Coherence Loss" to explicitly promote this structure. This loss function minimizes the maximum cross-correlation between filters:
$$L = MSE + \beta \cdot \|A^T A - |G_{ETF}|\|_\infty$$
This encourages the weights to behave like an ETF without forcing a rigid geometric constraint. This mechanism forces the network to learn redundant, robust representations; to be robust to the erasure of any specific neuron, the remaining neurons must be spread out in the feature space.
Experimental Results
The authors evaluated their ETF-promoting regularization on several standard benchmarks, including Fashion MNIST, CIFAR-10, Tiny ImageNet, and Penn Tree Bank (using LSTMs). They demonstrated consistent improvements in test accuracy when combining Dropout with their ETF regularization.
Key performance highlights include:
- Fashion MNIST: Accuracy improved from 90.16% (Dropout only) to 91.91% (Dropout + ETF).
- Tiny ImageNet (Top-1): Accuracy increased from 48.35% to 49.78%.
- LSTMs (Penn Tree Bank): Test perplexity dropped from 93.92 to 90.13.
Ablation studies confirmed that minimizing the maximum correlation (the $\ell_\infty$ norm) was more effective than minimizing average correlation ($\ell_1$ or $\ell_2$), as it focuses the optimizer on the most redundant filters.
Conclusion
This work suggests a deep relationship between Dropout and the ETF structure of regularized linear operations. While the computational cost of calculating the coherence loss is higher than standard Dropout, the geometric target it provides helps the optimizer find better global minima. Future work may involve exploring more efficient ways to apply these principles to the complex structures of convolutional layers.
Summary updated