Abstract
Dropout is a popular regularization technique in deep learning. Yet, the reason for its success is still not fully understood. This paper provides a new interpretation of Dropout from a frame theory perspective. By drawing a connection to recent developments in analog channel coding, we suggest that for a certain family of autoencoders with a linear encoder, optimizing the encoder with dropout regularization leads to an equiangular tight frame (ETF). Since this optimization is non-convex, we add another regularization that promotes such structures by minimizing the cross-correlation between filters in the network. We demonstrate its applicability in convolutional and fully connected layers in both feed-forward and recurrent networks. All these results suggest that there is indeed a relationship between dropout and ETF structure of the regularized linear operations.
Introduction
Dropout is a cornerstone of modern deep learning, effectively preventing overfitting by randomly "dropping" units during training. Despite its ubiquity, the underlying geometric constraints it imposes on network weights are not fully characterized. While often viewed through the lens of ensemble learning or Bayesian approximation, this work proposes a novel theoretical bridge between Dropout and frame theory. The authors argue that Dropout encourages weights to form Equiangular Tight Frames (ETFs)—geometric structures where filters are maximally spread apart—and that explicitly promoting these structures can significantly enhance performance.
Theoretical Framework: Dropout and Frame Theory
The researchers draw a parallel between a Denoising Autoencoder (DAE) using Dropout and the problem of analog channel coding in signal processing. In this scenario, a signal must be encoded such that it remains reconstructible even if some components are lost (erasures).
The ETF Connection
For linear encoders, the optimal solution for signal recovery from random erasures is an Equiangular Tight Frame (ETF). In an ETF, all vectors (filters) have equal length, and the angles between any two vectors are identical and maximized. This "spread" ensures that if some filters are dropped, the remaining ones maintain a robust representation of the input.
Coherence Regularization
Because finding a perfect ETF is a difficult, non-convex optimization problem, the authors introduce Coherence Loss ($CL$). This term explicitly minimizes the maximum cross-correlation (coherence) between filters in a layer. By using the $L_\infty$ norm (ETF max), the optimization focuses on the most similar pairs of filters—a form of "hard example mining" for weight separation.
Experimental Results
The approach was evaluated across several benchmarks, including Fashion MNIST, CIFAR-10, Tiny ImageNet, and the Penn Tree Bank dataset for LSTMs.
Key Results
The addition of ETF regularization consistently improved test accuracy across various architectures. Notably, the impact was most pronounced in fully connected layers, where the geometric structure of weights is less constrained than in convolutional layers.
As shown in the table above for Fashion MNIST, the ETF max ($\ell_\infty$) regularization yielded the highest accuracy, supporting the strategy of focusing on the most coherent filter pairs. In more complex tasks like Tiny ImageNet, combining Dropout with ETF regularization improved VGG-16 Top-1 accuracy from 39.92% to 49.78%. In LSTMs, the combined approach significantly reduced perplexity on the Penn Tree Bank dataset.
Discussion and Limitations
The findings suggest that Dropout acts as a geometric architect, implicitly pushing networks toward robust ETF-like structures. Explicitly regularizing for these structures serves as a powerful complement to Dropout.
However, several challenges remain:
- Computational Cost: Calculating the Gram matrix and its maximum off-diagonal element for coherence loss adds overhead.
- Optimization: Weights rarely reach the theoretical Welch bound due to the non-convex nature of the landscape.
- Convolutional Complexity: Current implementations regularize individual kernels; a more rigorous approach would involve the full Toeplitz matrix of the convolution.
- Non-linearity: While the linear case is theoretically grounded, extending this framework to deep, non-linear classification networks is an ongoing area of research.
Summary updated