Semi-supervised learning trains a model on a few labeled examples plus many unlabeled ones. A common version, pseudo-labeling, lets the model guess labels for the unlabeled data and then retrain on those guesses. Adding more unlabeled text to these pipelines eventually stops helping, and sometimes starts to hurt. Most existing defenses act one example at a time: they drop low-confidence guesses, soften the loss, or require two models to agree. A recent study of weak supervision found that per-example defenses of this kind reduce the damage once the model passes its "crossover point" but do not move that point (Xu et al.). The crossover point is where extra weak labels start to hurt rather than help.
We propose a different kind of defense, borrowed from older unsupervised alignment work. That work found that a word type, such as the string "bank" across a whole corpus, usually takes only a few labels. It also found that models fail when they spread probability over many spurious labels for rare words. We apply a sparsity penalty to the corpus-wide label distribution of each word type, computed from all of its pseudo-labeled occurrences together, and use it to correct the pseudo-labels before retraining. A per-example filter cannot see this pattern. A corpus-level constraint can, and its evidence grows as the unlabeled pool grows. Our hypothesis is that this lets structured semi-supervised taggers and parsers keep improving at unlabeled scales where they currently plateau.
Core Claim
Saturation in pseudo-label training for structured NLP is partly caused by label-support diffusion. As the unlabeled pool grows, each word type, and especially each rare type, accumulates a growing tail of distinct wrong labels or wrong head attachments. Each wrong label is individually confident enough to pass a threshold, but together they form a diffuse distribution. This is the "garbage collector" failure that Vaswani et al. identified in word alignment, here appearing inside self-training.
The claim makes two testable predictions. First, under token-level methods, the number of distinct labels per type should grow with unlabeled volume and track the accuracy plateau. Second, a soft, type-level smoothed $\ell_0$ penalty on pseudo-label distributions should hold that support roughly flat and push the plateau to larger unlabeled scales. The new element is the level at which sparsity is applied. Spa sharpens each sentence's tree distribution with temperature at one fixed unlabeled scale. We constrain the aggregate over the whole corpus and study the full scaling curve.
Research Program
The method reuses Vaswani et al., who penalize the number of nonzero translation parameters with a smoothed count. The difference is where we apply that penalty: they apply it to model parameters inside EM, and we apply it to pseudo-label posteriors between self-training rounds.
Each round has three steps. First, the current model produces a posterior $p_i(y)$ over labels $y$ for every unlabeled token $i$. Second, we find corrected posteriors $r_i$ by solving
$$\min_{r}\; \sum_i \mathrm{KL}(r_i \,\|\, p_i) \;+\; \beta \sum_{w} \sum_{y} \left(1 - e^{-q_w(y)/\epsilon}\right), \qquad q_w(y) = \frac{1}{c_w}\sum_{i \in w} r_i(y).$$
Here $w$ ranges over word types, $c_w$ is the number of unlabeled occurrences of $w$, and $q_w$ is the corrected corpus-level label distribution for type $w$. The term $1-e^{-q/\epsilon}$ is the smoothed count of nonzero entries, with $\epsilon$ setting the share below which a label counts as spurious. The weight $\beta$ trades fidelity to the model against sparsity. Third, the model is retrained on the corrected posteriors.
The KL term grows with $c_w$ while the penalty does not. As a result, a frequent type keeps a minority label whenever its token-level evidence is strong, which protects polysemous words such as "run." A rare type with scattered guesses is pushed toward one or two labels. The hard one-tag-per-type rule of Lee et al. cannot behave this way. We solve the problem with projected gradient descent on each type's simplex, as Vaswani et al. do. The problem separates by type, so it parallelizes over the vocabulary.
For POS tagging, $y$ is the tag. For dependency parsing, $y$ is the dependent type's (head tag, direction) pair. Posteriors over this pair come from TreeCRF arc marginals computed with the inside-outside algorithm. The correction becomes an additive bias on arc scores before the next round's targets are formed.
The analysis has two parts. We trace scaling curves of accuracy against $\log n_u$, where $n_u$ is the number of unlabeled sentences. We also run a support diagnostic: the mean number of labels holding more than 1% mass per type, broken down by frequency band, as a function of $n_u$. The causal question is whether the diagnostic and the plateau move together.
We do not claim to beat the information-theoretic floor that He et al. derive. Their floor depends on how strongly the pseudo-labels are coupled to the labeled data, and suppressing errors shared across examples is one route toward that floor rather than past it. Their theory assumes Gibbs learners, so here it motivates the work but does not constrain the design.
Baselines
Naive reference points. The first is the supervised-only model trained on the labeled seed. The second is plain self-training on the same unlabeled pools, with no filtering or correction. These two set the floor and show the raw saturation curve.
Token-level confidence thresholding. This is self-training that keeps only pseudo-labels whose maximum probability exceeds a cutoff. It is the most widely used way to sparsify pseudo-labels. We tune the cutoff on the same dev set and give it the same budget of tuning runs as $\beta$ and $\epsilon$.
Spa. Spa combines virtual adversarial training on a TreeCRF with temperature sharpening. It is the strongest existing method that makes semi-supervised structured distributions sparse. It reports 92.82 UAS (unlabeled attachment score) with 10% of PTB labeled and BLLIP unlabeled text, against 92.11 for its supervised baseline. We reproduce Spa on its 10% split with the same encoder and the same BLLIP sentences at each scale. Beating it means two things: exceeding its UAS at the largest pool, and showing a curve that is still rising where Spa's has flattened.
For full-data parsing, the dense-feature, billion-token system SS-SCM reports 93.79% on English WSJ. We use it as a reference for how far unlabeled text alone has been pushed, not as a matched competitor, because its supervision and compute differ.
For semi-supervised POS tagging from a small labeled seed with a neural tagger, no reported result was found on Lacuna. The closest number, 93.4% for the LDA model of Toutanova and Johnson, is a dictionary-supervised setting, and we cite it only as context.
Ablations isolating the key ingredient. The first ablation applies the same smoothed-$\ell_0$ penalty to each token's posterior individually. This keeps the sparsity form and drops the corpus aggregation, so it tests whether aggregation is what matters. The second is a hard one-tag-per-type projection, which tests whether softness matters. The third is an entropy-raising regularizer in the style of WLSR (Chen et al.), which pushes distributions in the opposite direction.
Why not just…?
Why not just use confidence thresholding, which already discards uncertain guesses? Thresholding judges each token alone. A rare word can receive five different wrong tags across five sentences, each at 0.97 confidence, and every one survives the filter. The per-token $\ell_0$ ablation and the thresholding baseline test this directly. If either matches the type-level method on rare-type accuracy and on the support diagnostic, our mechanism is wrong.
Why not just lower the temperature, as Spa does? Temperature sharpens within a sentence but does not coordinate across sentences. Spa was evaluated at only one unlabeled scale, so it is untested whether its gains persist as the pool grows. If Spa's curve keeps rising as fast as ours, the corpus-level constraint adds nothing. This threat is real, and the scaling comparison settles it.
Why not just use a strong pretrained encoder, which may make pseudo-labels clean enough? This is the most serious threat. With good contextual representations, the benefit of extra unlabeled text may be small to begin with. We therefore run every condition with and without a pretrained encoder. We report honestly if diffusion disappears in the pretrained regime, which would narrow the claim to low-resource languages and weak encoders. One prediction survives even then: rare and out-of-vocabulary types should still show diffusion.
Why not just smooth labels, which prevents overconfidence? Smoothing raises entropy everywhere, including on labels that are impossible. Meister et al. show that label smoothing is an extreme setting of a family of entropy regularizers, and that variants allowing exact zeros did better. They report 32.0 against 31.7 BLEU on WMT'14 De-En. The WLSR-style baseline tests the opposite direction directly in the self-training loop.
Proposed Experiments
POS tagging. We use WSJ with labeled seeds of 1k sentences, 5k sentences, and the full training set. Unlabeled BLLIP sentences scale from $10^4$ through roughly $6.5 \times 10^5$, and then into English Gigaword as compute allows. CoNLL-X Spanish and Portuguese, from Lee et al., test whether the effect holds across languages.
Parsing. We use Spa's 10% PTB split with the same BLLIP pool, extended to larger scales. Each point is run with three seeds.
Decisive checks. There are four. First, the plateau must shift: the type-level method keeps a positive slope against $\log n_u$ beyond the scale where thresholding and Spa flatten. Second, the support diagnostic must grow with $n_u$ for the baselines but stay flat for our method, with the gap concentrated in low-frequency types. Third, accuracy on genuinely ambiguous types must not fall below plain self-training; we define these as gold types with a second tag holding at least 10% of occurrences. Fourth, a crossover-style test borrowed from Xu et al.: we increase the labeled seed until self-training stops helping, and ask whether that labeled size moves to the right under the type-level constraint.
A failure on the first or second check, with the per-token ablation matching the full method, would falsify the claim that corpus aggregation is what drives the gain.
References
- From Theory to Decision Rule: Calibrating the Noisy-Label Crossover for VLM Weak Supervision Across Three Medical-Imaging Benchmarks
- Smaller Alignment Models for Better Translations: Unsupervised Word Alignment with the l0-norm
- Spa: On the Sparsity of Virtual Adversarial Training for Dependency Parsing
- Simple Type-Level Unsupervised POS Tagging
- How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?
- An Empirical Study of Semi-supervised Structured Conditional Models for Dependency Parsing
- A Bayesian LDA-based model for semi-supervised part-of-speech tagging
- Semi-supervised feature learning for improving writer identification
- Generalized Entropy Regularization or: There's Nothing Special about Label Smoothing