Spending Speech Labels Where the Coarse Model Is Sure: Broad-Phonetic Filtering for Active Learning in Noisy Audio

Active learning for speech recognition picks which unlabeled recordings to send to human transcribers by asking where the current model is least sure. The trouble is that a speech model is also unsure about audio that nobody can learn from, such as clipped, babble-soaked, or near-silent segments. Pure uncertainty sampling therefore spends its budget on noise. We propose a cheap fix that comes from an old idea in phonetics. Phonetic alphabet reduction collapses dozens of phones into a handful of broad classes, such as vowels, stops, nasals, and fricatives. Broad classes stay recognizable long after fine distinctions are lost. Our score asks two questions of every frame: is the model sure about the broad class, and unsure which phone inside that class it is? Frames that pass both tests point to learnable gaps in the model's knowledge. Frames that are uncertain even at the broad level are treated as likely noise and down-weighted.

The two-sentence version is this. Split the recognizer's uncertainty into "which broad sound class?" and "which phone within the class?" Then label the utterances where the first is confident and the second is not. The split costs nothing extra, because broad-class probabilities are just sums of phone probabilities. It needs no beam search and no repeated stochastic forward passes. Whether it beats speaker-regularized confidence sampling and dropout-based uncertainty on noisy pools is the empirical question this proposal is built to answer.

Core Claim

Let $Y_t$ be the model's predicted phone or grapheme at frame $t$. Let $B_t = g(Y_t)$ be its broad class under a fixed many-to-one map $g$; the blank symbol used by CTC, the alignment-free training loss common in end-to-end speech models, is treated as its own class. Because $B_t$ is a deterministic function of $Y_t$, the model's per-frame predictive entropy splits exactly:

$$H(Y_t) = H(B_t) + H(Y_t \mid B_t).$$

Here $H(B_t)$ is computed from the marginalized posterior $p(b) = \sum_{y: g(y)=b} p(y)$. The term $H(Y_t \mid B_t) = \sum_b p(b)\, H(Y_t \mid B_t=b)$ is the expected within-class uncertainty. Standard least-confidence sampling effectively ranks on the left-hand side. We rank utterances $u$ by

$$s(u) = \frac{1}{|u|}\sum_{t \in u} H(Y_t \mid B_t)\,\mathbf{1}[H(B_t) < \tau] \;-\; \lambda\, \rho_\tau(u).$$

In this score, $\tau$ is a broad-entropy threshold and $\rho_\tau(u)$ is the fraction of frames in $u$ whose broad entropy exceeds $\tau$. The weight $\lambda$ sets how strongly broad-uncertain, likely-corrupted utterances are pushed down the list.

The claim is that $s(u)$ spends a smaller share of the labeling budget on unrecoverable audio than least-confidence sampling. It should also do no worse on clean pools, and it should approach the benefits of dropout-based separation of learnable from unlearnable uncertainty at a fraction of the compute.

Why This Is Not Already Done

Two literatures meet here without touching. Alphabet reduction is used as a training-time regularizer with fixed data. Reduce and Reconstruct trains on a 27-symbol reduced alphabet for Gujarati and Telugu and rebuilds full orthography with finite-state transducers. Verma et al. maps code-mixed scripts into a shared reduced space. The noisy-WSJ studies of Fosler-Lussier et al. and Hartmann et al. reduce unstressed syllables to manner or place classes. All four use static training sets and never ask which data to label.

Active learning for speech works on full alphabets. Malhotra et al. rank by 1-best path probability with a speaker-diversity penalty. Riccardi et al. use decision-boundary distance on alphadigit HMMs. Neither separates noise-driven uncertainty from learnable uncertainty. The closest active-learning work in phonetics, Bootstrapping phonetic lexicons, is text-only and scores confidence with string similarity rather than model posteriors.

The papers play three roles in this proposal. Motivation comes from Peng et al., who show that maximum-uncertainty selection chases unlearnable samples and that median-uncertainty selection trains more stably. A constraint comes from Yen-Ju Lu et al.. On TIMIT at 0 dB SNR, they report phone accuracy near 16% while broad-class accuracy stays near 40%. That gap is what makes the decomposition plausible.

It is also a warning, and we address it head on. Even at 0 dB the broad level is only partly reliable, so heavily corrupted frames may look "broad-confident, fine-uncertain" and slip through as false learnable signal. Settling this is the first experiment, not an assumption.

Research Program

We build the score on a CTC recognizer whose output layer covers the full phone or grapheme set. The broad map $g$ comes in three versions:

  • Manner map. The six manner classes of Huttenlocher et al., applied to TIMIT and English corpora.
  • Expert reduced map. The 27-symbol reduction from Reduce and Reconstruct, applied to Gujarati and Telugu.
  • Learned map. Clusters of the seed model's own confusion matrix, matched in size to the expert maps.

We compare two ways of getting broad probabilities. The first marginalizes the fine posterior, which needs no new parameters. The second adds a separate broad-class head trained jointly, since a dedicated head might be better calibrated under noise.

The active-learning loop starts from a small labeled seed. It scores the pool with greedy, frame-level posteriors only, with no beam search. It selects a fixed number of hours per round, retrains, and repeats. As a secondary efficiency question, we test whether scores taken from an intermediate encoder layer preserve the ranking, in the spirit of SIFT. SIFT computed entropy from BERT's first layer to skip half the backward passes.

Baselines

The naive reference is random selection at the same per-round hours, with the same seed, model, and training schedule.

The strongest existing method is Regularized Least Confidence from Malhotra et al.. It is 1-best path probability from beam decoding, combined with i-vector speaker clustering to avoid sampling the same speakers repeatedly. They report a 9.8% relative WER reduction and 10% relative CER reduction over random selection at 20 hours on LibriSpeech, and about a 12.5% relative CER reduction on the Corpus of Spontaneous Japanese. It is the strongest competitor because it is the only speech-specific method in the file that already corrects a known failure of plain confidence sampling.

We will rerun it on our pools, not quote it. It gets the same seed, the same retraining recipe, and the same tuning budget for its diversity weight as we give $\tau$ and $\lambda$. Because speaker diversity is orthogonal to our noise filter, we also run our score with the same speaker penalty added. That keeps the comparison about the uncertainty term alone.

Beating Regularized Least Confidence means two things. On noise-contaminated pools, our score must give lower WER at matched labeled hours across at least three budgets, with differences outside seed variance over three runs. On clean pools, it must stay within seed variance.

The second baseline is plain least confidence, the same scoring without the speaker penalty.

The third is median-uncertainty selection, adapted from Peng et al.: rank by full-alphabet entropy and take utterances nearest the median rather than the top. No reported result found on Lacuna for median-uncertainty sampling in ASR.

The fourth is Monte Carlo dropout mutual information, the disagreement across stochastic forward passes, which is the textbook way to isolate learnable uncertainty. We use the K=30 passes setting from Wu et al.. No reported result found on Lacuna for MC-dropout active learning in ASR.

Four ablations isolate the key ingredient:

  • No mask. Rank by $H(Y_t \mid B_t)$ without the broad-confidence mask. This tests whether the filter, not the decomposition, does the work.
  • Random map. Replace $g$ with a random grouping of the same size. Reduce and Reconstruct found random reductions significantly worse for training. We expect the same for selection if phonetic structure matters.
  • Learned map. Swap the expert maps for confusion-matrix clusters of the same size.
  • Separate head. Use a dedicated broad-class head instead of marginalization.

Why not just…?

Why not just use Monte Carlo dropout, which already separates learnable from unlearnable uncertainty? This is the most serious threat. Dropout disagreement could do everything our score does. Our case rests on two points. First, it costs K forward passes per utterance over the whole pool, while our score costs one greedy pass. Second, dropout disagreement on badly corrupted audio need not be low, because the model has seen little such audio, so it may still rank noise highly. The experiment settles this directly. If dropout mutual information matches or beats our score at equal labeled hours, the remaining claim reduces to compute. We will then report wall-clock selection cost and whether our score reaches the same WER curve at a fraction of that cost. We will not claim an accuracy advantage we did not find.

Why not just filter out low-SNR audio with an SNR estimator and run ordinary least confidence? On synthetic noise at known SNR, an oracle SNR filter is a strong, simple competitor, and we include it as an upper reference. Its weakness is that corruption is not one number. Clipping, codec loss, overlapping speech, and disordered or accented speech can be learnable or not regardless of SNR. Clipping is the case where Cosi found allophone merging prevented fitting noise. Our evaluation pool therefore mixes corruption types, some of which an SNR estimate misjudges. If the SNR filter ties us everywhere, the phonetic decomposition adds nothing and we will say so.

Why not just add speaker diversity, as Regularized Least Confidence already does? Speaker diversity fixes repeated speakers, not wasted labels on unintelligible audio. Malhotra et al. explicitly do not address the noise problem. This is why we also run the combined method. If the speaker penalty alone closes the gap on noisy pools, our premise is wrong.

Proposed Experiments

Experiment 1: does broad entropy separate corrupted frames from clean ones? This diagnostic comes first and is decisive. On TIMIT, which has time-aligned phone and broad labels, we add noise at SNRs from 20 dB down to 0 dB, as in Yen-Ju Lu et al.. We then measure how well $H(B_t)$ separates corrupted from clean frames, compared with how well total entropy $H(Y_t)$ separates them. We also check whether the masked within-class term flags clean frames that the seed model gets wrong. If broad entropy does not separate noise better than total entropy, the method fails here and we stop.

Experiment 2: noisy English pools. We build a LibriSpeech pool where a controlled fraction of utterances (0%, 20%, 40%) is corrupted with car noise, factory noise, babble, and clipping. These are the noise types studied on WSJ by Fosler-Lussier et al. and Hartmann et al.. Starting from a 20-hour seed, we run all baselines to 60 hours. We report three things: WER on clean and noisy test sets per labeled hour, the share of selected hours drawn from corrupted utterances, and selection wall-clock time.

Experiment 3: low-resource Indian languages. We repeat the loop on Gujarati and Telugu from the Microsoft Speech Corpus with the expert 27-symbol map. We stop at the 10-hour regime where Reduce and Reconstruct reports 4.8% to 7% relative WER gains from reduction alone. This tests whether reduction helps twice, once in training and once in choosing the data.

What would sink the proposal. Any of three outcomes would count against it: broad entropy failing the frame-level separation test, dropout mutual information matching it at equal labeled hours, or the random-map ablation performing as well as the phonetic map. Any of these would mean the phonetics is not doing the work.