Abstract
Dropout is a cornerstone of modern neural network training, effectively creating an ensemble of exponentially many submodels through random neuron deactivation. Standard inference typically employs uniform weight scaling, multiplying weights by the dropout probability to approximate the average submodel output. This approach, rooted in a "bagging" perspective, assumes all submodels are equally reliable. In this work, we challenge this assumption, proposing that submodels exhibit significant heterogeneity. We introduce a non-uniform weight scaling method that optimizes a layer-wise scale vector post-training. Our results on CIFAR-10 and Fashion-MNIST show that this "boosting" approach consistently reduces error rates by better aggregating the diverse sub-structures learned during dropout training.
Introduction
Dropout is widely used to prevent overfitting by randomly "dropping" neurons during training. This process can be viewed as training an ensemble of $2^n$ submodels that share parameters. At test time, the challenge lies in aggregating these models efficiently. The standard "Weight Scaling Inference Rule" (WSIR) uses a single global factor—the dropout probability $p$—to scale weights. This assumes that every random configuration of the network is a "low-bias" learner. However, in practice, factors like limited data or early stopping can lead to a mix of high-bias and overfit submodels. We propose shifting from this "bagging" mindset to a "boosting" mindset, where submodels are weighted according to their performance.
Methods
We propose replacing the scalar $p$ with a scale vector $\mathbf{s} = [s_1, s_2, ..., s_n]$ corresponding to the input dimensions of a layer. The optimal $\mathbf{s}$ is found by solving a constrained optimization problem: minimize the loss on a subset of data while keeping the mean of $\mathbf{s}$ equal to $p$. This ensures the total activation energy remains consistent with the training phase. We utilize a reparameterization trick and a Lagrangian formulation to optimize $\mathbf{s}$ using standard gradient-based methods (e.g., Adam) after the primary model weights are frozen.
Experimental Results
We evaluated our method on CIFAR-10 and Fashion-MNIST using residual networks with dropout placed before the final output layer. Non-uniform scaling consistently outperformed both uniform scaling and Monte Carlo (MC) sampling across multiple trials.
| Dataset | Uniform Scaling (Error) | Monte Carlo (Error) | Non-uniform Scaling (Error) |
|---|---|---|---|
| CIFAR-10 (Test) | 9.98% ± 0.20 | 9.98% ± 0.22 | 9.83% ± 0.25 |
| Fashion-MNIST (Test) | 8.37% ± 0.25 | 8.37% ± 0.25 | 8.18% ± 0.23 |
Analysis of Scale Distributions
A key finding is that the optimized scale vectors are not uniform. As shown in the histograms below, the scales for both CIFAR-10 and Fashion-MNIST spread significantly around the default value. This variance confirms that the model benefits from weighting certain features or sub-structures more heavily than others during inference.
Conclusion
Non-uniform weight scaling provides a robust, post-training refinement for dropout-regularized models. By acknowledging submodel heterogeneity, we can extract better performance from existing architectures without increasing inference-time complexity. Future work will explore the application of this method to deeper layers and its interaction with various training hyperparameters.
Summary updated