When a large pretrained network is fine-tuned on a few dozen or a few hundred labeled examples, gradient descent tends to latch onto a handful of strongly predictive features and ignore the weaker, redundant ones. Those ignored features are often what makes the model hold up on data from a new hospital or a new camera. A recent result shows that a very blunt fix works: switch off 90% of the penultimate-layer features at random during fine-tuning, which forces the classifier to spread its weight across many of them. Learnable dropout methods are the obvious next step. These methods treat each feature's drop probability as a trainable parameter instead of a fixed setting, so a natural guess is that they should do better than one uniform rate. This note argues that, as usually built, they should do worse in this regime, and proposes a version that should do better.
The argument is simple. Learnable dropout rates are normally fit to training loss, either alone or plus a prior term. Training loss rewards keeping exactly the dominant features that gradient descent already prefers and dropping the weak ones. Learned rates therefore reinforce the bias that very large dropout exists to break. We propose to learn per-feature dropout rates from out-of-fold loss instead. The small training set is split into folds. The network weights are updated on some folds, and the dropout rates are updated by how well the network does on the held-out fold. The rates then get credit only for noise patterns that help prediction on unseen examples. The claim is testable and could fail: if a single uniform rate chosen by the same cross-validation budget matches it, the per-feature machinery is not worth having.
Core Claim
In small-sample fine-tuning, what matters is which features receive noise, not only how much noise there is in total. The signal for learning that allocation has to come from held-out examples, not training examples. We predict three things. First, Concrete dropout fit by its usual objective will learn low drop rates on high-gradient features and will underperform a tuned uniform very-large dropout. Second, cross-fitted rates will match or beat the best tuned uniform rate, with the largest gains when the target data come from a different distribution than the pretraining data. Third, the cross-fitted rates will be measurably anti-correlated with how much the classifier relies on each feature during ordinary training. That third prediction is the mechanism. The second is the payoff.
Where This Sits
The proposal draws on three groups of prior work, and each group plays a different role.
The first group motivates the proposal. Fine-tuning with Very Large Dropout applies a fixed 90% dropout to the penultimate representation of a ResNet-50 during fine-tuning. It argues that this counters stochastic gradient descent's implicit preference for a few dominant features, and it reports 74.0% average out-of-distribution accuracy on DomainBed against 72.0% for plain empirical risk minimization. Its rate is a global constant, so it cannot ask whether some features should be dropped more than others. Our method keeps its placement (the penultimate layer, during fine-tuning) and its diagnosis (redundant features are underused), and replaces the constant with a learned per-feature profile.
The second group constrains the design. Explore the Power of Dropout on Few-shot Learning and its companion A Unified Framework with Meta-dropout for Few-shot Learning find that dropout applied during few-shot target fine-tuning can hurt, and that it helps when applied to the backbone during base-class pretraining. This looks like it contradicts the very-large-dropout result. Our reading is that both findings concern uniform rates. A uniform rate that destroys the few features a 1-shot task depends on hurts. A uniform rate that pushes a 100-example task toward redundant features helps. Learned per-feature rates are a direct way to test this reading. The same finding also warns us that at 1 shot there is nothing to hold out, so we restrict the claims to regimes with at least a handful of examples per class.
Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation is a second constraint. It shows that Concrete dropout (dropout rates made trainable by replacing the binary mask with a smooth relaxation) is practical on a 41-scan dataset and localizes errors well. It is also overconfident on out-of-distribution tumors. We do not claim to fix out-of-distribution detection, and we will not evaluate it as a headline result.
The third group supplies methods and the main principled competitor. Adaptive Dropout with Rademacher Complexity Regularization and Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm also try to give dropout rates an out-of-sample signal. They do this through a theoretical generalization bound added to the training loss, with weights and rates updated in alternation. The second paper reports a jump on the Wine dataset from 62.8% with fixed 0.5 dropout to 97.8%, but only for shallow networks. The first notes that its bound needs heuristic scaling in larger models. The two approaches differ in where the out-of-sample signal comes from. Theirs comes from a worst-case bound. Ours comes from measured held-out loss, which needs no scaling heuristic but costs data and compute. The variational line, from Variational Dropout and the Local Reparameterization Trick through Variational Dropout Sparsifies Deep Neural Networks, supplies the low-variance gradient machinery we reuse. It also illustrates the failure we predict: under its objective, weak features are pushed toward being pruned entirely, which is exactly the opposite of what out-of-distribution robustness needs. Contextual Dropout makes rates depend on each input. We deliberately do not do this, because per-input rates add parameters that a 100-example dataset cannot support.
Research Program
The method has three components.
The first is the dropout layer itself. Let $\Phi(x)\in\mathbb{R}^d$ be the penultimate features of a pretrained backbone for input $x$, and let $p_j$ be the drop probability of feature $j$. During training, each feature is multiplied by a relaxed Bernoulli (Concrete) mask, which keeps the mask differentiable with respect to $p_j$.
The second is cross-fitting, which supplies the held-out signal. The $N$ training examples are split into $K$ folds. At each step, one fold $k$ is sampled to serve as the held-out set. The weights $w$ take a gradient step on the other folds, $D_{-k}$, using the dropout-perturbed loss. The rates are then updated using the gradient of the clean, mask-free loss on fold $k$, computed through that one-step lookahead:
$$ p \leftarrow p - \eta_p \nabla_p \, \mathcal{L}_{D_k}\big(w - \eta_w \nabla_w \mathcal{L}^{\text{drop}}_{D_{-k}}(w, p)\big). $$
Here $\mathcal{L}_{D_k}$ is the loss on fold $k$ without dropout, $\mathcal{L}^{\text{drop}}_{D_{-k}}$ is the loss on the remaining folds with masks drawn at rates $p$, and $\eta_w$ and $\eta_p$ are step sizes. Rotating the held-out fold means every example contributes to training the weights. The rates receive gradient only through generalization to examples the current step did not train on. The extra cost is roughly one additional forward and backward pass per step.
The third component is a parameterization that keeps the method usable at tiny $N$. With $d=2048$ free rates and 50 examples, the rates themselves could overfit the held-out folds. We therefore tie rates into groups of features clustered by their correlation on the training set, and initialize all rates at 0.9 so the method starts from the very-large-dropout solution and only has to learn deviations from it. A diagnostic sweep varies the number of groups from 1 to $d$. With one group, the method reduces to learning a single uniform rate by cross-validation.
The analysis tests the mechanism directly. For each feature, we record its reliance under plain fine-tuning, measured as the classifier's weight magnitude multiplied by the feature's variance. We then correlate reliance with the learned retention under three schemes: training-loss Concrete dropout, Rademacher-bound adaptive dropout, and cross-fitted dropout. The prediction is a positive correlation under the training-loss scheme and a negative one under cross-fitting.
Baselines
The naive reference point is plain fine-tuning with no dropout on the penultimate layer. On full DomainBed, Fine-tuning with Very Large Dropout reports 72.0% average out-of-distribution accuracy for this baseline. We will rerun it under our subsampled protocol, for which no reported result was found on Lacuna.
The strongest existing method is very-large dropout itself: a uniform rate on the penultimate layer, with that paper's 74.0% DomainBed average as the published anchor. To make the comparison fair, the uniform rate will not be fixed at 0.9. It will be chosen from {0.5, 0.7, 0.8, 0.9, 0.95} by the same $K$-fold split on the same training examples. The backbone, augmentation, number of epochs, and total compute will match the proposed method, with the uniform baseline allowed roughly twice as many epochs to absorb our extra pass. Beating it means a higher mean out-of-distribution accuracy across at least three of the four DomainBed datasets at each subsample size, with paired seeds and an improvement larger than the seed-to-seed standard deviation.
The principled-rate competitors are Concrete dropout trained with its standard objective, and Rademacher-regularized adaptive dropout from Adaptive Dropout with Rademacher Complexity Regularization. Both get the same rate tying and the same initialization, and the Rademacher method's bound-scaling coefficient is tuned on the same folds. These test whether held-out loss is a better signal than training loss or a theoretical bound.
Two ablations isolate the key ingredient. The first takes the learned rate profile and randomly permutes it across features, which preserves the mean and the distribution of rates but destroys which feature gets which rate. The second sets every feature to the learned mean rate. If either ablation matches the full method, the per-feature allocation is doing no work.
Why not just…?
Why not just tune one uniform dropout rate by cross-validation? This is the real threat. On a few hundred examples, a single scalar may capture most of what is learnable, and the very-large-dropout paper's results on VLCS show broad plateaus where rates of 0.90 and 0.95 behave similarly. The proposal is designed so that this alternative is a special case: it is the one-group setting of our method. If the group sweep shows no gain beyond one group, together with the permuted-profile and mean-rate ablations, we will report that per-feature rates are unnecessary in this regime. That would be a useful negative result, because it would settle a design question that both the dropout and the few-shot literatures leave open.
Why not just use Concrete dropout or variational dropout, which already learn rates? Their objectives reward fitting the training data. In Variational Dropout Sparsifies Deep Neural Networks, this drives low-relevance weights toward removal, which is the wrong direction when weakly relevant features carry robustness. On tiny datasets, the prior term also dominates in a way that depends on $N$ rather than on any measure of generalization. Our reliance-correlation analysis tests this mechanism directly, and including Concrete dropout as a baseline tests whether it matters for accuracy.
Why not just use a smaller network, linear probing, or deep ensembles? Unveiling COVID-19 from Chest X-ray with deep learning shows that ResNet-18 beats ResNet-50 on small chest X-ray data, so shrinking capacity is a genuine competitor. We will include a ResNet-18 run and a frozen-backbone linear probe. Deep ensembles had the best Dice score on Spleen in the uncertainty benchmark, at a multiple of the memory and compute, so we will report an ensemble of five plain fine-tuned models as a cost-unmatched upper reference rather than as a like-for-like baseline. If the ensemble wins outright, our method's case rests only on cost.
Proposed Experiments
The main testbed is DomainBed (PACS, VLCS, OfficeHome, TerraIncognita) with a ResNet-50, using the very-large-dropout protocol. The source-domain training sets are subsampled to 16, 64, and 256 examples per class, and models are evaluated on the held-out domain. The subsampling is what creates the small-sample regime. It also means the published numbers serve only as full-data anchors, so every method is rerun under our protocol.
The clinical test is CORDA, a set of 447 chest X-rays. We use the lung-cropped ResNet-18 pipeline of Unveiling COVID-19 from Chest X-ray with deep learning as the base model, with training on public data and testing on CORDA. That paper reports a sensitivity of 0.54 and a specificity of 0.80, and that is the bar to clear. Because shortcut learning on scanner artifacts is the known failure in this setting, lung cropping stays in place for every method. Our method is meant to sit on top of preprocessing, not to replace it.
A secondary check is 5-shot fine-grained classification on CUB, to see where the regime boundary lies. Only 1-shot numbers are reported in the file (63.27% for Baseline++ and 69.05% with meta-dropout), and no reported result was found on Lacuna for 5-shot CUB with these methods. We also expect the method to give no benefit at 1 shot, because a single example per class cannot be split into folds.
The decisive checks are of three kinds. Cross-fitted rates must beat the cross-validated uniform rate in matched comparisons, and the permuted-profile ablation must lose that advantage. Training-loss Concrete dropout must fall below the tuned uniform rate, with positive reliance-retention correlation, while the cross-fitted method shows negative correlation. Gains must grow as the training subsample shrinks from 256 to 16 examples per class, and as the gap between source and target domains widens; TerraIncognita is expected to show the largest effect and PACS the smallest. If the first check fails, the per-feature idea is wrong. If the first passes but the second fails, the method may work for a reason other than the one proposed, and we would say so.
References
- fine-tuning with very large dropout
- Explore the Power of Dropout on Few-shot Learning
- A Unified Framework with Meta-dropout for Few-shot Learning
- Benchmarking Scalable Epistemic Uncertainty Quantification in Organ Segmentation
- Adaptive Dropout with Rademacher Complexity Regularization
- Deep Metric Learning: The Generalization Analysis and an Adaptive Algorithm
- Variational Dropout and the Local Reparameterization Trick
- Variational Dropout Sparsifies Deep Neural Networks
- Contextual Dropout: An Efficient Sample-Dependent Dropout Module
- Unveiling COVID-19 from CHEST X-Ray with Deep Learning: A Hurdles Race with Small Data