Label-Degree Spectra: Why Sequence Losses Break Under Noisy Transcripts

Speech recognizers are increasingly trained on transcripts that a machine produced, and those transcripts contain wrong words. Discriminative sequence losses are the objectives that align audio to a transcript and push probability away from competing transcripts; the standard example is CTC (connectionist temporal classification, a loss that sums over all ways of aligning a transcript to the audio frames). These losses break down badly on such data. On LibriSpeech with half the tokens substituted, plain CTC reaches 36.1% word error rate, while a method that lets the model skip suspect tokens reaches 7.4% (Bypass Temporal Classification). The fixes that work are heuristic: skip-token arcs with a hand-tuned penalty schedule, latent-transcript beam search running on CPU, or confidence filtering. None of them explains why one wrong token does so much damage, or predicts which objective will degrade fastest.

This note proposes a measurable explanation borrowed from the analysis of Boolean functions. Fix one utterance and one audio frame. The gradient that the loss sends to that frame can be viewed as a function of which transcript tokens are corrupted, and this function has a Fourier expansion over subsets of tokens. A term that depends jointly on $d$ tokens keeps exactly a $(1-2\eta)^d$ fraction of its clean value when each token is independently corrupted with rate $\eta$. So objectives whose gradients rely on high-order interactions among tokens, as alignment-based losses plausibly do, should lose signal exponentially faster than objectives whose gradients are local. We propose to measure this "label-degree spectrum" directly, test whether it predicts degradation across objectives, and use it to set a cheap smoothing of the gradient that suppresses the parts noise has already destroyed.

Core Claim

Let $b \in \{-1,1\}^L$ mark which of the $L$ transcript tokens are corrupted ($b_i=-1$ means token $i$ was substituted, with the replacement token averaged out). Let $g(b)$ be the loss gradient at a fixed frame and vocabulary entry, expanded as $g(b)=\sum_{S\subseteq[L]}\hat g(S)\chi_S(b)$ with $\chi_S(b)=\prod_{i\in S}b_i$. The clean gradient is $g(\mathbf{1})=\sum_S \hat g(S)$. Under independent corruption at rate $\eta$, $\mathbb{E}[b_i]=1-2\eta$, so

$$\mathbb{E}_{b\sim\mu_\eta}[g(b)] = \sum_{S\subseteq[L]} (1-2\eta)^{|S|}\,\hat g(S),$$

where $\mu_\eta$ is the corruption distribution. This identity is exact. It is the Bonami–Beckner noise operator $T_{1-2\eta}$, evaluated at the clean labeling. The degree-$d$ part of the gradient keeps a $(1-2\eta)^d$ fraction of its signal on average, and the rest becomes variance.

The empirical claim has three parts. First, CTC and lattice-style discriminative losses concentrate gradient mass at higher label degree than frame-level cross-entropy on a fixed alignment, because one substitution reroutes the whole alignment and changes gradients at distant frames. Second, the shape of each objective's degradation curve as $\eta$ grows is predicted by its measured spectrum on clean data. Third, bypass arcs work because they cap each token's influence and so shift mass to low degree.

The constructive consequence is a gradient estimator that applies $T_\rho$ to the label side, with $\rho$ chosen from the measured spectrum and the noise level. It averages the loss over $K$ random re-corruptions of the transcript while reusing a single encoder pass. This damps high-degree components, which under noise carry mostly variance, and leaves low-degree components nearly intact.

How the Three Directions Fit

The discrete–continuous work supplies the operator and its interpretation. Smoothed Agnostic Learning of Halfspaces over the Hypercube shows that random bit flips act through the Bonami–Beckner operator as a discrete stand-in for Gaussian smoothing, damping high Fourier degrees so that low-degree regression suffices. That work smooths inputs to a single halfspace and needs the noise level to be known. We apply the same operator to label-corruption bits of a sequence loss, and we estimate the needed smoothing from data. The invariance principle of Noise stability of functions with low influences says that low-influence multilinear functions of bits behave like the same functions of Gaussians. Here it justifies modeling the aggregated high-degree residual as roughly Gaussian noise when no single token dominates. That noise model is what makes a shrinkage (Wiener-style) choice of $\rho$ principled. It is also where we expect the theory to fail, since a few high-influence tokens break low-influence regularity.

The Boolean sensitivity work supplies the link between influence and degree. Degree and Sensitivity: tails of two distributions proves that a function of sensitivity $s$ is $\epsilon$-approximated in $L_2$ by degree $O(s\log(1/\epsilon))$. Applied to our gradients, it predicts that objectives bounding how many frames a single token flip can change will have spectra truncated near low degree, so smoothing costs them little clean signal. The Average Sensitivity of an Intersection of Half Spaces is a constraint rather than a tool. It warns that average sensitivity and noise sensitivity can diverge, as in its tribes counterexample. We therefore measure noise stability directly and do not infer it from per-token influence. For measuring spectra near a small corruption rate rather than at 50%, we use the biased-measure Fourier analysis of Keller et al..

The speech work motivates the question and supplies the testbeds. The degradation of MPE-style training on machine transcripts in Yu, Gales and Woodland and the U-shaped confidence curve in Huang et al. are the phenomena to explain. Lead2Gold marginalizes over latent clean transcripts inside the log; this denoises but needs CPU beam search. Bypass Temporal Classification adds a skip-token arc to every token, governed by the brittle schedule $\lambda_i=\beta\tau^i$. Our estimator does neither. It averages the loss outside the log over corruptions, which is exactly $T_\rho$, and costs $K$ extra loss evaluations on shared encoder outputs.

Research Program

The first component is a spectrum estimator. For a model checkpoint and an utterance, we draw pairs of $\rho$-correlated corruption patterns under a $\eta_0$-biased measure. We recompute only the loss and its gradient with respect to the encoder logits, using k2 on GPU as in the BTC setup, and estimate the noise stability $\sum_S \rho^{|S|}\hat g(S)^2$ at several values of $\rho$. The degree profile follows by fitting the resulting polynomial in $\rho$ (a moment method). We also record an influence-by-distance profile: how much flipping token $j$ changes the gradient at frames far from where token $j$ aligns. This tests the non-locality hypothesis directly. The objectives we profile are frame cross-entropy on a fixed forced alignment (expected to be local and low-degree), CTC, BTC, and, where feasible in k2, lattice-free MMI.

The second component is the prediction test. From clean-data spectra alone, and before training on noisy data, we compute each objective's predicted signal-retention curve $\sum_d (1-2\eta)^d W_d$, where $W_d$ is the measured gradient weight at degree $d$. We then compare this curve to observed error rate as a function of $\eta$.

The third component is the noise-matched smoothing estimator. We pick $\rho$ to minimize a plug-in estimate of the squared distance between the smoothed noisy gradient and the clean gradient, computed from the measured profile and an estimate of $\eta$. The estimate of $\eta$ comes either from a small audited subset, in the spirit of directed manual transcription, or from a blind sweep. The adaptive variant re-estimates the profile periodically during training. This addresses the known-noise-level limitation of the smoothed-learning result.

Baselines

The naive reference is standard CTC trained on the noisy transcripts with no correction. BTC reports 36.1% WER for it at 50% substitution on LibriSpeech train-clean-100, and 63.1% phone error rate at 70% substitution on TIMIT. On clean data CTC reaches 5.6% WER. The smoothed estimator must not raise this clean number beyond noise, because the theory says low-degree signal is preserved.

The strongest existing competitor on this exact question is Bypass Temporal Classification. It reports 7.4% WER at 50% substitution on LibriSpeech and 21.4% phone error rate at 70% substitution on TIMIT, and it directly caps per-token influence, which is the mechanism our theory credits. We rerun it in k2 with the same encoder, data, noise injection, and step budget. It receives the same number of tuning trials for $\beta$ and $\tau$ that our method receives for $\rho$ and $K$. Beating it means lower error at matched budget in at least the high-noise conditions, or equal error without the collapse failures BTC shows under mistuned schedules. If BTC wins outright, the fallback contribution is showing that smoothing stacks on top of BTC.

Lead2Gold is the principled latent-variable competitor. It reports 46.4% error on dev-clean under its hardest "M30 f2" condition, against 87.7% for the ASG baseline. We reproduce its WSJ-weak-model noise generation on train-clean-100 and compare error at matched wall-clock time, since its CPU beam search is its main cost.

Two ablations isolate the key ingredient. The first is single-sample random transcript corruption at the same $\rho$, which is an unbiased but high-variance estimate of the same operator. The second is $K$-sample smoothing with $\rho$ tuned by grid search rather than set from the spectrum. Together they separate the value of variance reduction from the value of the theory-guided choice of $\rho$.

Why not just…?

Why not just randomly corrupt transcripts as data augmentation? In expectation, one random corruption per step is our estimator, so this is the main threat. Our method differs in two ways: it averages $K$ corruptions over a shared encoder pass, and it derives $\rho$ from the measured spectrum. If the single-sample ablation with a grid-tuned rate matches our method at equal encoder compute, we will report that the method adds nothing beyond augmentation. The contribution then reduces to the diagnostic: predicting which objectives degrade and which augmentation rate to use without a sweep.

Why not just filter by confidence? Huang et al. found that the most confident data is redundant and the least confident is harmful, with the best gains from the middle: 15.0% relative WER reduction from importance sampling versus 11.7% from random sampling. Filtering removes whole utterances, whereas errors are token-level. We also test whether per-utterance label-degree weight explains the U-shape, since middle-confidence utterances might combine novelty with low-degree gradients.

Why not just early-stop on a clean development set? This is a strong and simple control. We include it for every objective. It does not explain why objectives differ, and it requires clean held-out data, which low-resource settings such as the 16-hour Indonesian set in Leveraging Label Preprocessing may lack. We report whether smoothing improves the best early-stopped checkpoint rather than only the final one.

Why not just use BTC with a better schedule? If our spectrum shows that BTC already sits at low degree, the theory predicts that smoothing adds little on top of it. That is a falsifiable prediction we test, and confirming it would itself account for BTC's success.

Proposed Experiments and Decisive Checks

The main testbeds are LibriSpeech train-clean-100 and TIMIT with synthetic substitution noise from 10% to 70%, following BTC. We add insertion and deletion noise, which BTC handles poorly and which change the length $L$, so the corruption-bit model must be extended to edit operations. We also use the Lead2Gold M-series conditions, which are generated by a weak WSJ model and so resemble real recognizer errors. For naturally noisy labels we use the Indonesian MCV/FLEURS 16-hour wav2vec 2.0 setup, where unnormalized training gives 33.45% WER and the 5-step normalization gives 15.83%. We ask whether smoothing closes part of the remaining gap after normalization.

The first decisive check is predictive. Spectra computed on clean data must rank the objectives' degradation curves correctly, and the retention curve must correlate with observed error across the $\eta$ sweep. If frame cross-entropy, CTC, and BTC do not order by label degree, the core explanation is wrong.

The second check is mechanistic. Influence-by-distance profiles should show CTC gradients responding to flips of distant tokens more than fixed-alignment cross-entropy does.

The third check is constructive. At matched encoder compute, noise-matched $K$-sample smoothing should beat both plain CTC and single-sample augmentation, with $\rho$ from the spectrum landing near the grid-search optimum.

The fourth check is on BTC. Smoothing added to BTC should give gains smaller than those it gives CTC, in proportion to BTC's lower high-degree weight.

The metrics are word error rate, phone error rate, and character error rate, the degree profiles themselves, and wall-clock cost per step.