When a classifier is trained on a handful of labeled examples, its fitted weights are off in a consistent direction, not just noisy. Maximum-likelihood training (ordinary cross-entropy) has a systematic error that shrinks roughly as one over the number of examples, and it makes predictions overconfident. A classical statistical fix, Firth's penalized likelihood, cancels this leading error term by adding a penalty to the loss. In deep few-shot classification and low-budget active learning, the fix helps reliably. In practice, though, the strength of the penalty, a coefficient $\lambda$, is chosen with a labeled validation set. That is exactly the resource these regimes do not have. The theory that motivates the penalty also prescribes how strong it should be, and how that strength should fall as data accumulates. Nobody in this literature has checked whether that prescription holds up empirically.
This note proposes to derive the coefficient Firth's theory implies for the classifier heads used in practice, to test it against tuned and learned coefficients, and to see whether it can replace validation-based tuning. If the theory-set coefficient matches tuned performance, small-sample bias reduction becomes free of extra labels. If it fails, the measurements will show where the first-order theory stops describing deep classifiers, which is itself an empirical result this literature currently lacks.
Core Claim
Firth's correction adds $\tfrac{1}{2}\log|I(\theta)|$ to the summed log-likelihood, where $I(\theta)$ is the Fisher information of the classifier parameters $\theta$. Its weight relative to the data term is therefore fixed by the theory rather than chosen freely. Written in the per-example (mean) loss that deep learning code actually minimizes, the correction becomes
$$ \mathcal{L}(\theta) = \frac{1}{N}\sum_{i=1}^{N} \ell_{\text{CE}}(x_i, y_i;\theta) \;-\; \frac{1}{2N}\log|I(\theta)|, $$
where $N$ is the number of labeled examples and $\ell_{\text{CE}}$ is per-example cross-entropy.
On the Importance of Firth Bias Reduction in Few-Shot Classification shows that for multinomial logistic and cosine heads, this penalty takes the form of a KL divergence pulling predictions toward the uniform distribution, $\mathcal{L}_{\text{CE}} + \lambda D_{\text{KL}}(U\,\|\,P_\theta)$. That paper treats $\lambda$ as a hyperparameter tuned on validation data. Our claim is that carrying the theoretical weight through this equivalence yields a coefficient
$$ \lambda_F(N) = \frac{\kappa}{N}, $$
where $\kappa$ is computed from the head's structure (number of classes, feature dimension, and any cosine temperature) and is not tuned. The $1/N$ form comes directly from dividing the fixed-weight penalty by the $N$ terms in the mean loss.
This makes two testable predictions. First, $\lambda_F$ should land near the validation-optimal $\lambda$ at each $N$. Second, the across-round decrease that Taming Small-sample Bias in Low-budget Active Learning learns with bilevel optimization should approximately follow $1/N$. CHAIN, the method in that paper, also learns a within-round increase in $\lambda$ across gradient steps, which the first-order theory does not predict. We will test whether that component carries real signal or mainly compensates for optimization dynamics.
Research Program
The first stage is analytic. We will derive $\kappa$ exactly for the logistic and cosine heads covered by the Firth equivalence, using frozen pretrained features so the theory applies cleanly. Where the KL form is only an approximation to $\tfrac{1}{2}\log|I|$, we will also implement the exact log-determinant penalty, which is cheap for a linear head on modest feature dimensions, so that any gap between the two can be attributed correctly.
The second stage directly measures bias, which the deep few-shot papers never do because the true parameters are unknown. On frozen features from a pretrained backbone, we will fit a "teacher" logistic head on a large labeled pool and then sample synthetic labels from it. This gives a setting where the true $\theta^*$ is known. We will then estimate the mean error of the learned head versus $N$ under five conditions: plain cross-entropy, $\lambda_F$, validation-tuned $\lambda$, and two classical label-free corrections, jackknife and parametric bootstrap. Averaging over thousands of resampled labeled sets, we will report the bias slope against $1/N$ and the residual after correction. That residual indicates how large the uncharacterized $O(N^{-2})$ terms are in practice.
We will validate the measurement code first on a case with a published answer. Maximum likelihood estimation of the Weibull distribution with reduced bias reports a shape-parameter bias of 2.251 for plain maximum likelihood versus −0.007 for its analytic correction, at $n=10$, 30% uncensored, and true shape 10. Our code must reproduce these values before we trust it on classifier heads.
The third stage replaces tuned or learned $\lambda$ with $\lambda_F(N)$ in the two downstream settings where the Firth penalty already has documented gains. We put more weight on active learning, because there the validation set is a direct cost against the labeling budget. Few-shot classification is a secondary check.
Baselines
The do-nothing reference is plain cross-entropy fine-tuning with no bias penalty, run on identical labeled sets, backbones, and optimizer settings. Every method must show a gain over this reference, and the gain should shrink as $N$ grows if the effect is truly small-sample bias.
The strongest existing competitor is CHAIN from Taming Small-sample Bias in Low-budget Active Learning. It learns a dual schedule for $\lambda$ by bilevel optimization against a validation set. On CIFAR-10 with 10 queries per round, it reports raising final accuracy from approximately 52% to 60%. We will run CHAIN in two versions. The first follows the original protocol, with an external validation set; we treat this as an upper reference. The second is the fair version, where the validation labels are drawn from the same annotation budget as everything else, so every method spends the same number of labels. Both versions use the same backbone (ResNet-18 fine-tuning and logistic regression on frozen features), the same query strategies (Random, Entropy, CoreSet, BADGE), and the same seeds. For few-shot classification, the main competitor is the validation-tuned static $\lambda$ of On the Importance of Firth Bias Reduction in Few-Shot Classification, which reports absolute gains of 0.5% to 2.5% over plain training on mini-ImageNet, tiered-ImageNet, and CIFAR-FS.
The label-free statistical competitors are jackknife bias correction of the head weights and leave-one-out cross-validation of $\lambda$ on the labeled set itself. Both are given the same total compute as our method.
Two ablations isolate the key ingredient. One fixes $\lambda$ at the value $\lambda_F(N_0)$ from the first round and never decays it, which tests whether the $1/N$ law matters. The other adds CHAIN's within-round increase on top of $\lambda_F$, which tests whether that component adds anything.
We count our method as beating the strongest competitor under three conditions. It must match CHAIN with external validation within seed-to-seed confidence intervals. It must significantly exceed budget-matched CHAIN at 10 and 20 queries per round. And $\lambda_F$ must fall within a factor of two of the validation-optimal $\lambda$ across the range of $N$ we test.
Why not just…?
Why not just tune $\lambda$ on base-class episodes, which cost no novel labels? In few-shot classification this is a real threat, and it may work as well as $\lambda_F$. We will test it directly. However, it only transfers if the optimal $\lambda$ at the novel task's $N$ matches the one tuned at the base episodes' $N$. Our second prediction says it should not: the optimum should move as $1/N$. If transferred tuning matches $\lambda_F$ at every $N$, we will report that the theory adds nothing in the few-shot setting, and the contribution narrows to active learning. In cold-start active learning, there is no base dataset from the same task to tune on.
Why not just use the jackknife, which removes the one-over-$N$ bias without any tuning? Like Firth, the jackknife cancels the first-order term, and it is the most natural label-free alternative. Its weaknesses are cost, since it requires $N$ refits, and higher variance at very small $N$. The bias-measurement stage compares the two head to head on bias, variance, and mean squared error against known $\theta^*$. If the jackknife is equally good and affordable at the label counts we study, we will say so.
Why not just use label smoothing or temperature scaling to fix the overconfidence? Both shrink predictions toward uniform, as the Firth KL form does. On the Importance of Firth Bias Reduction in Few-Shot Classification reports that Firth's penalty outperformed label smoothing and L2 regularization in its experiments. Label smoothing still has a strength to tune, and post-hoc temperature scaling needs held-out labels. We include both, tuned by leave-one-out cross-validation, as secondary baselines.
Proposed Experiments
The decisive active-learning experiment runs on CIFAR-10, CIFAR-100, and Fashion-MNIST at 10 and 20 queries per round across all four query strategies, with at least ten seeds and paired t-tests, following CHAIN's protocol. The key comparison is $\lambda_F$ against budget-matched CHAIN. We also overlay the learned CHAIN schedule against $\kappa/N$.
Few-shot evaluation runs 1-shot and 5-shot tasks on mini-ImageNet, tiered-ImageNet, CIFAR-FS, and the mini-ImageNet→CUB cross-domain split, with logistic and cosine heads. The comparison is against the validation-tuned static $\lambda$ and against base-class-transferred $\lambda$.
The bias-measurement experiment reports the error of the head weights against $1/N$ for each correction. Alongside accuracy, it reports expected calibration error, since overconfidence is the stated symptom.
The idea fails cleanly if the validation-optimal $\lambda$ does not scale as $1/N$, or if it sits far from $\kappa/N$. In that case, the first-order Firth theory does not describe these heads well, and the residual-bias curves will show by how much.