Adaptive Dropout with Rademacher Complexity Regularization

Ke Zhai, Huan WangICLR 2018 · OpenReview

In the world of deep learning, Dropout is a household name. It’s the classic technique of randomly "switching off" neurons during training to prevent them from relying too heavily on each other (co-adaptation), which helps the model generalize better to new data. But there has always been a nagging question: What should the dropout rate be?

Usually, researchers pick 0.5 for hidden layers and 0.2 for inputs based on "rules of thumb," or they run expensive grid searches to find the "sweet spot." A new paper from ICLR 2018, "Adaptive Dropout with Rademacher Complexity Regularization," proposes a more elegant solution: let the model learn its own dropout rates using a solid mathematical foundation.

TL;DR

Instead of manually tuning dropout rates, this paper derives a theoretical bound on a neural network's complexity (Rademacher complexity) that explicitly includes dropout probabilities. By treating this bound as a regularizer in the loss function, the model automatically learns optimal, neuron-specific dropout rates that improve performance and offer surprising insights into what the model finds important.

What Problem Does This Paper Address?

The standard approach to dropout has two major flaws:

  1. Heuristic Tuning: Selecting the "retain rate" (the probability a neuron stays active) is often a matter of trial and error.
  2. Coarse Granularity: Standard dropout applies the same rate to every neuron in a layer. This assumes all features are equally prone to overfitting, which is rarely true.

The authors sought to move dropout from a "heuristic trick" to a theoretically justified component of the objective function. They wanted a system where the model could say, "This specific neuron is very noisy; I should drop it more often," or "This feature is critical; I should keep it active."

Methods

The researchers turned to Rademacher Complexity, a measure of a function class's ability to fit random noise. Generally, a model with high Rademacher complexity is more likely to overfit.

The Core Innovation: Theorem 3.1

The authors proved a new theorem showing that the Rademacher complexity of a neural network is bounded by a function of the weight matrices and the vector of dropout retain rates.

Trade-off between empirical loss and the Rademacher complexity regularizer across different retain rates.

From Theory to Training

By adding this complexity bound to the standard cross-entropy loss as a regularizer, they created a unified objective function. They then used a Block Coordinate Descent algorithm to optimize the model:

  • Step A: Fix the dropout rates and update the network weights (standard backprop).
  • Step B: Fix the weights and update the dropout rates to minimize the complexity-weighted loss.

To keep training efficient, they used a "deterministic approximation" during the dropout rate updates—essentially using the expected value of the neurons rather than sampling thousands of random masks.

Datasets

The authors evaluated their method across several standard benchmarks:

  • Image Classification: MNIST (handwritten digits), CIFAR-10, and CIFAR-100 (object recognition).
  • Text Classification: SUBJ (subjectivity analysis) and IMDB (sentiment analysis).

Key Insights and Ideas

  • Neuron-Wise Adaptivity: The model learns a unique retain rate for every single neuron. This allows for much finer control over regularization than layer-wide settings.
  • The "Aha!" Moment in Visualization: When the authors visualized the learned rates for MNIST, they found that pixels in the center (where the digits are) had high retain rates, while pixels on the edges (background noise) were dropped almost entirely.
Visualization of learned retain rates for MNIST pixels, showing high retention in the center and dropping background noise.
  • Semantic Importance in Text: In sentiment analysis (IMDB), the model learned to keep "opinion" words like perfect, worst, and love with high probability, while dropping neutral words like year or music. The math effectively performed feature selection automatically.
  • Stability: Despite starting with different random weights, the dropout rates tended to converge to similar distributions across multiple runs, suggesting the model is finding a robust structural property of the data.

Key Results

The "Rademacher" approach consistently outperformed standard dropout and other adaptive methods like Variational Dropout:

  • MNIST: Reduced error to 0.95% on a 3-layer network, compared to 1.25% for standard dropout.
  • CIFAR-100: Achieved a classification error of 38.63%, significantly better than the 41.00% achieved by standard dropout.
  • Text: On the SUBJ dataset, it achieved 10.15% error, outperforming all other dropout variants tested.

Limitations and Future Work

While the results are impressive, the authors noted a few areas for growth:

  • Heuristic Scaling: To prevent the regularizer from overwhelming the loss as networks get deeper or wider, they had to introduce some empirical scaling factors (heuristics) to the theoretical bound.
  • Sparsity: The current method doesn't strictly force retain rates to zero. The authors are interested in exploring "sparse" retain rates to potentially prune networks entirely during training.
  • PAC-Bayes Bounds: They suggest that moving from Rademacher complexity to PAC-Bayes frameworks might yield even tighter theoretical guarantees in the future.

By bridging the gap between statistical learning theory and practical deep learning, this paper moves us one step closer to "self-tuning" models that understand their own complexity.

Summary updated

Abstract

We propose a novel framework to adaptively adjust the dropout rates for the deep neural network based on a Rademacher complexity bound. The state-of-the-art deep learning algorithms impose dropout strategy to prevent feature co-adaptation. However, choosing the dropout rates remains an art of heuristics or relies on empirical grid-search over some hyperparameter space. In this work, we show the network Rademacher complexity is bounded by a function related to the dropout rate vectors and the weight coefficient matrices. Subsequently, we impose this bound as a regularizer and provide a theoretical justified way to trade-off between model complexity and representation power. Therefore, the dropout rates and the empirical loss are unified into the same objective function, which is then optimized using the block coordinate descent algorithm. We discover that the adaptively adjusted dropout rates converge to some interesting distributions that reveal meaningful patterns.Experiments on the task of image and document classification also show our method achieves better performance compared to the state-of the-art dropout algorithms.
BibTeX
@inproceedings{zhai2018adaptive,
  author = {Ke Zhai and Huan Wang},
  title = {{Adaptive Dropout with Rademacher Complexity Regularization}},
  booktitle = {International Conference on Learning Representations},
  year = {2018},
  url = {https://openreview.net/forum?id=S1uxsye0Z},
}
Figure