Matching Discriminative Capacity to Label Count in Generative Classifiers

A classifier can be built two ways. It can model how inputs are produced within each class and then apply Bayes' rule: this is a generative classifier, and it learns usefully from few labels. Or it can model the label given the input directly: this is a discriminative classifier, which wins once labels are plentiful. Most hybrids of the two blend their training losses with a fixed weight. We propose a different control knob: the number of parameters that are allowed to be trained discriminatively. A full-size generative classifier is trained first. Then only a small, nested set of correction parameters is tuned to predict labels, and that set grows as labeled data grows. The aim is a single model that follows the better of the two curves at every data size.

This matters because a recent from-scratch study of Transformers found an awkward pattern. The classic "generative wins small, discriminative wins large" crossover appears only in 12-layer models. At one layer, the discriminative encoder wins at every data size (Generative or Discriminative? Revisiting Text Classification in the Era of Transformers). Practitioners therefore have to guess both the model family and the model size in advance. We argue the pattern has one cause. A generative model's errors come mostly from a few systematic biases, and fixing them takes only a handful of label-trained parameters. If that is right, a correctly sized discriminative correction should rescue small generative models and protect large discriminative ones from overfitting.

Core Claim

Let $p_\theta(x \mid y)$ be a class-conditional generative model. Here it is an autoregressive Transformer (a model that predicts each token from the previous ones) that reads a label token first. It is trained by likelihood on the available text. We then define a ladder of nested correction parameters $\phi_k$, where $k$ is the number of trainable parameters at rung $k$. The corrected posterior is

$$ q_k(y \mid x) \propto \exp\Big( b_y + \sum_{g=1}^{G} w_{y,g}\, \bar\ell_g(x, y;\ \theta + \Delta_k) \Big). $$

In this posterior, $b_y$ are class biases. Each $\bar\ell_g$ is the length-normalized log-likelihood of the tokens that fall in group $g$. Groups are position buckets or token-frequency buckets, a learned version of the subject/body "regions" in earlier work. The weights $w_{y,g}$ rescale each group. The term $\Delta_k$ is a low-rank update to the generative weights whose rank grows only on the upper rungs. The ladder runs as follows. Rung 0 is the pure Bayes rule. The next rungs add class biases, then per-group weights, then rank-1, 2, 4, … updates to the attention layers. The top rung is full discriminative fine-tuning. The correction parameters are always trained to maximize $\log q_k(y\mid x)$.

The rung is chosen per dataset by cross-fitted conditional likelihood on the labeled set, so no separate validation split is needed. Our crude estimation argument predicts that the chosen size $k^*$ should grow roughly as the labeled count $N$ grows. The argument has two pieces. The discriminative estimation error scales like $\sqrt{k/N}$, as for any $k$-parameter conditional model. The bias removed by each additional rung shrinks.

The bold prediction is specific. On one-layer Transformers with small $N$, a low-rung hybrid should beat the discriminative encoder that dominated in Generative or Discriminative?. On 12-layer models, the hybrid should match the generative model at small $N$ and the best discriminative or masked-language-model variant at large $N$.

Relation to Prior Work

The method comes from one paper. Classification with Hybrid Generative/Discriminative Models trains Naive Bayes word probabilities by counting. It then tunes only an $R$-dimensional vector of region weights by leave-one-out conditional likelihood, and proves label requirements scale with $\log V$, where $V$ is the vocabulary size. We reuse its core move: a large generative part, a tiny discriminative part, and leave-one-out tuning. We extend it in three ways it leaves open. The generative part is a deep Transformer. The regions are learned groupings rather than hand-defined ones. Most importantly, the discriminative part is a ladder that grows with $N$ rather than a fixed $R$.

Other hybrids constrain the design. The Classification Restricted Boltzmann Machine (Learning Algorithms for the Classification Restricted Boltzmann Machine), Deep Hybrid Models (Hybrid Deep Discriminative/Generative Models for Semi-Supervised Learning) and the contrastive energy-based hybrid HDGE (Hybrid Discriminative-Generative Training via Contrastive Learning) all share every parameter between a discriminative and a generative loss, balanced by a fixed weight $\alpha$. The Deep Hybrid Models paper itself notes that this allocation is static. Our proposal never trades the two losses on the same parameters. It decides which parameters see labels at all.

Prior work motivates the proposal. Generative or Discriminative? supplies the depth-dependent crossover we try to explain, and also the warning that pretraining erases it. We therefore stay in the from-scratch regime, where its finding holds. The broader capacity literature shows smaller discriminative models beating larger ones on small or noisy data. Examples are StockMixer, Temporal Convolution Networks for Real-Time Abdominal Fetal Aorta Analysis with Ultrasound and Neural network language models for low resource languages. That literature sizes the whole model by trial and error. Our claim is that only the label-trained portion needs sizing, and that the size can be read off the labeled data. Towards Bridging Sample Complexity and Model Capacity bounds the capacity needed in terms of data structure. We use it as a lens for interpreting $k^*$ rather than as a method.

Research Program

Generative models. We train class-conditional autoregressive Transformers at 1, 4 and 12 layers on each task, following the from-scratch protocol of Generative or Discriminative?. In a semi-supervised variant, the unlabeled text also trains the label-marginalized likelihood.

The ladder. We implement the full correction ladder as a single parameterization with nested masks. The same code can then produce every rung, and every competing method shares one backbone and one tokenizer.

Analysis. We map the chosen rung $k^*$ against $N$ and depth. We fit its growth rate and check whether it relates to the dominating-cluster count from Towards Bridging Sample Complexity and Model Capacity, computed in the generative model's feature space.

Diagnosis. We test the bias hypothesis directly. If one-layer generative failure is mostly fixable bias, a few rungs should recover most of the gap between the generative model and the discriminative encoder. If the failure is spread throughout the representation, recovery will require the top rungs, and the hypothesis fails.

Baselines

Naive reference points. The do-nothing reference is the rung-0 generative classifier, the autoregressive model with Bayes rule. Alongside it we run the plain discriminative encoder at the same depth. Together these trace the two curves the hybrid must beat at their respective ends. Under the same protocol, Generative or Discriminative? reports qualitatively that the encoder wins at one layer and the generative model wins at low data for 12 layers. The file gives no scalar weighted-F1 for these models, so the paper's numbers cannot serve as a target. We regenerate them ourselves.

The strongest existing method. At high data and 12 layers, the strongest competitor is the masked-language-model classifier from the same paper. It achieved the best weighted-F1 across the nine datasets, but no reported scalar result was found on Lacuna for it. We rerun it with identical data, tokenization, and a matched hyperparameter-search budget.

Fixed-weight hybrid. The strongest hybrid competitor shares all parameters between cross-entropy and generative likelihood with weight $\alpha$, as in the ClassRBM and HDGE. HDGE reports 96.7% CIFAR-10 accuracy against 95.8% for its supervised baseline (Hybrid Discriminative-Generative Training via Contrastive Learning), but that is not our domain. We tune $\alpha$ with exactly the same number of trials that our method spends choosing a rung.

Text-classification anchor. On 20 Newsgroups, the reference is the Sparse Hybrid ClassRBM at 23.8% error against 32.8% for an SVM (Learning Algorithms for the Classification Restricted Boltzmann Machine). Our split and preprocessing may differ, so we rerun an SVM and a ClassRBM ourselves and treat the published figure only as a sanity check.

Ablations. Three ablations isolate the key ingredient. The first trains the same $k$ parameters with the generative loss instead of the conditional one. The second uses a random $k$-dimensional subspace instead of the structured ladder. The third is full fine-tuning (the top rung) with early stopping, which is what a practitioner would do by default.

What counts as beating them. Beating a baseline means a statistically reliable win over seeds and label subsamples at the label counts where that baseline is the curve to beat. Overall success means being within noise of the lower envelope of all baselines at every $N$ and depth, while no single baseline achieves that.

Why not just…?

Why not just use a small discriminative encoder? At one layer this is exactly what Generative or Discriminative? found works best. If our low-rung hybrid cannot beat it at small $N$, our central hypothesis is wrong, and we will report that. The capacity literature also says the right size depends on the task and must be found by search. Our method instead adapts size through $k^*$ within one trained model.

Why not just tune the loss weight $\alpha$? The fixed-weight hybrid is the direct alternative and is run with a matched search budget. The difference is structural. Under a shared-parameter blend, every parameter still sees the noisy label gradient at small $N$. On our ladder, at small $N$ the labels touch only a handful of parameters. If a tuned $\alpha$ matches us everywhere, the ladder adds nothing, and the comparison will show it.

Why not just pretrain generatively and fine-tune everything with early stopping? This is the real threat, because early stopping is itself an implicit capacity limit. The top-rung ablation tests it directly. We expect it to lose at the smallest $N$, where a few hundred labels cannot reliably pick the stopping point. We also expect it to be worse calibrated there. If it ties, the contribution shrinks to an interpretable capacity schedule, and we would say so.

Why not just use pretrained models? Generative or Discriminative? shows that pretraining makes discriminative encoders dominate. Our claims are therefore restricted to settings without a suitable pretrained model, such as new domains, scratch-trained small models, or latency-limited deployment. We do not claim the result transfers.

Proposed Experiments

Data and grid. We use the nine text benchmarks of Generative or Discriminative?: AG News, Emotion, SST-2, SST-5, Multiclass Sentiment, Twitter Financial News, IMDb, and Hate Speech Offensive. We subsample labels on a logarithmic grid from about 32 per class to the full set. We run depths 1, 4 and 12 with five seeds and five label subsamples per cell. We add 20 Newsgroups to connect to the region-weighting and ClassRBM results.

Metrics. We measure weighted-F1, expected calibration error, accuracy under injected label noise, and inference latency. All are metrics the reference study already uses.

Decisive checks. The proposal stands or falls on four checks. First, the cross-fitted choice $k^*$ must rise with $N$ on most datasets; a flat $k^*$ means the ladder is not tracking anything. Second, at one layer and small $N$, a hybrid from the bottom few rungs must beat the discriminative encoder. Third, at every depth and $N$, the hybrid must stay within noise of the best baseline, which no single baseline manages. Fourth, the generative-loss and random-subspace ablations must fall clearly below the structured, conditionally trained ladder at matched $k$. That result shows the gain comes from label-trained, well-placed correction parameters rather than from parameter count alone.