Hindsight Teachers for Federated Training of Speech Rescorers

Speech recognizers on phones and smart speakers usually work in two passes. A fast first pass produces a short ranked list of candidate transcripts, the "N-best list." A heavier second model, the "rescorer," then re-ranks that list using broader linguistic context. The strongest rescorers are trained discriminatively: they learn directly to put the lowest-error candidate on top. That requires the true transcript, and the true transcript is exactly what a user's device never has. Federated learning trains a shared model by sending weight updates, not raw data, from many devices to a server. It could in principle personalize rescorers to how real users talk. In practice there has been no label signal on the device worth learning from.

We propose that the label signal is already on the device, and it arrives late. After a conversation or dictation session ends, the device holds the N-best lists for every utterance in the session, including the ones that came after any given utterance. A rescorer that reads the whole session in both directions, a "hindsight teacher," can often resolve an ambiguous word using what the speaker said next. The streaming rescorer used at inference time can only see the past. We distill the hindsight teacher's judgments over each N-best list into the compact streaming rescorer, on the device, while it is idle. Only small adapter updates are shared through federated averaging. Because the teacher has strictly more information than the student, this is not the circular self-training that makes pseudo-labeling fragile. It is learning from privileged information that exists only after the fact.

Core Claim

A causal (past-context-only) discriminative rescorer can be trained federatedly, with no transcripts on any client, by matching its distribution over each N-best list to that of an on-device teacher that also sees future utterances in the same session. The resulting model should beat unigram-level federated personalization and self-training baselines on conversational and meeting speech, at no added inference latency.

For utterance $u$ in session $s$, let $\mathcal{H}_u$ be its N-best list. Let $p_T(h \mid \mathcal{H}_u, s)$ be the teacher's softmax over candidates $h \in \mathcal{H}_u$, conditioned on the full session. Let $p_S(h \mid \mathcal{H}_u, s_{<u})$ be the student's softmax, conditioned only on earlier utterances. Each client minimizes a confidence-weighted distillation loss:

$$\mathcal{L}_i = \sum_{u \in D_i} c_u \, \mathrm{KL}\big(p_T(\cdot \mid \mathcal{H}_u, s) \,\|\, p_S(\cdot \mid \mathcal{H}_u, s_{<u})\big)$$

Here $D_i$ is client $i$'s set of utterances. The weight $c_u$ is a confidence that is high when the teacher's distribution is sharp and changed from the first-pass ranking only because of future context. This is the N-best posterior cross-entropy of Distillation Strategies for Discriminative Speech Recognition Rescoring, which compressed a 170M-parameter centralized teacher into a 5M student. We reuse their loss. We replace their oracle-based correction, which needs transcripts, with a confidence weight that does not.

Relation to Prior Work

The one paper in this space that connects federated learning and rescoring is Federated Marginal Personalization for ASR Rescoring. It keeps a neural language model frozen and multiplies its word probabilities by a factor built from personal, federated-global and background unigram distributions. Those distributions are estimated from first-pass hypotheses, with Laplace noise added for utterance-level privacy. It reports 4.8% relative WER reduction on AMI and 2.4% on Curated Video. This paper both motivates and bounds our proposal. It shows that unlabelled N-best lists carry usable personalization signal. It is also limited to single-word frequency shifts, never trains the network, never optimizes ranking directly, and leaves user-level privacy open. Our method trains the rescorer's parameters on a ranking objective, and we will report user-level privacy.

The supervised discriminative rescorer we build on is RescoreBERT. It turns a bidirectional encoder into a single-pass scorer over the [CLS] token and trains it with MWER (minimum expected word error rate over the N-best list) and MWED (matching word-error distributions). On LibriSpeech it reports 6.6% relative WER reduction on test-clean and 3.4% on test-other over a non-discriminative BERT rescorer. It is fully centralized and needs ground truth. We keep its architecture and its focus on the N-best distribution, and swap the ground-truth target for the hindsight teacher.

Session-level Language Modeling for Conversational Speech supplies the motivation that cross-utterance context matters: 3–5% relative WER gains on conversational telephone speech from past turns alone. It also documents a constraint we exploit: errors in first-pass history mislead later rescoring. A hindsight teacher can revise history, so the context it feeds itself is cleaner than the student's.

Three federated speech systems constrain the design by showing what does and does not work without labels. FedNST trains acoustic students on hard 1-best pseudo-labels from a server teacher and throws away the alternatives in the N-best list. Federated Self-Learning with Weak Supervision uses an exponential-moving-average copy of the student as teacher, with implicit user feedback as reward, and needs rehearsal on cloud data to limit forgetting. That teacher sees the same information as the student. Ours sees more. Parameter-Efficient Transfer Learning under Federated Learning for ASR shows that communicating only bottleneck adapters (4–8% of a Conformer) matches centralized adaptation on Fleurs EN-US, but with source-domain degradation. We adopt adapters for the rescorer and measure forgetting explicitly. All three adapt the first pass. None trains the rescorer.

Research Program

The work has four parts.

Build the models. We build a compact 5M-parameter causal rescorer in the RescoreBERT style. It scores each candidate from the candidate's text plus an encoding of the previous utterances' top hypotheses. We also build a larger bidirectional session teacher, initialized from the centralized RescoreBERT-style model. The teacher scores each candidate using the top hypotheses of both preceding and following utterances. It is itself iterated: it re-rescores the whole session, then conditions on its own revised history.

Train federatedly. The teacher runs on-device only offline, for example while charging. Its cost does not count toward inference latency. Clients compute the loss above and send adapter updates for the student. Updates are aggregated by FedAvg with per-user clipping and Gaussian noise for user-level differential privacy, the setting studied for end-to-end ASR in Federated Learning with Differential Privacy for End-to-End Speech Recognition. The teacher is frozen during federated rounds. In a second phase, periodically refreshing the teacher from the aggregated student lets us test whether gains compound or collapse.

Analyze where the signal comes from. We measure how often the hindsight teacher's top choice differs from the causal model's. When it differs, we measure how often it is right, using labelled held-out data. This "hindsight flip precision" determines whether the method can work at all.

Characterize the conditions. We map the regime where the method helps against session length, speaker count, and the oracle WER of the N-best list. As Large and Diverse Language Models for Statistical Machine Translation and LIMSI @ WMT12 note, the oracle WER caps any rescorer.

Baselines

First-pass only. The do-nothing reference is the first-pass recognizer's 1-best output. A slightly stronger version adds a centralized rescorer trained with MWER on public labelled data and never adapted. Every federated method must beat this second system, or federation is pointless.

Federated Marginal Personalization. This is the strongest existing competitor, because it is the only published method that personalizes a rescorer federatedly from unlabelled N-best lists. It reports 4.8% relative WER reduction on AMI over a background neural LM rescorer. We will re-implement it on the same first-pass N-best lists, the same client partition, and the same number of rounds. We tune its interpolation weights and kernel bandwidth with the same hyperparameter budget as our method. Since we cannot guarantee our first pass matches theirs, success means beating our own re-run of it on the same lists, not the quoted 4.8%. We also run it stacked with our method, since the two are compatible.

Causal self-training. This ablation isolates the key ingredient. It uses an identical federated pipeline, but the teacher is an exponential-moving-average copy of the causal student, or the same large teacher restricted to past context only. It has the same compute, adapters, rounds and privacy noise. If hindsight is the active ingredient, this variant should give markedly smaller gains or drift over rounds.

Centralized supervised upper bound. RescoreBERT-style MWER fine-tuning on the client data with transcripts, pooled centrally, bounds what any label-free federated method can reach. Recovering a large fraction of the gap between the first-pass-plus-centralized-rescorer system and this upper bound counts as success.

First-pass federated adaptation. FedNST-style pseudo-label training of the acoustic model, with a frozen rescorer, tests whether adapting the first pass makes rescorer adaptation redundant.

Why not just…?

Why not just use marginal (unigram) personalization? Unigram factors capture which words a user says often, such as names and jargon. They cannot capture the errors that dominate conversational speech: homophones and phrase ambiguities resolved by context, like "their" vs "there" or a name heard earlier. The unigram-only limitation is stated in the paper's own design. The real threat is that on short sessions, vocabulary is most of the available gain. We will report results stratified by session length. If our method only matches marginal personalization on short sessions but wins on long ones, we will say so.

Why not just self-train the rescorer on its own top choices? A model trained on its own rankings reinforces its own mistakes. That is the confirmation bias FedNST fights with a server teacher and seed labels, and that Rao et al. counter with rehearsal and user-feedback rewards. The causal self-training ablation above is the direct test. Our claim fails if it performs as well as the hindsight teacher.

Why not just run the session teacher at inference time? For offline transcription such as meeting minutes, one can, and the proposal adds nothing there beyond personalization. For voice assistants and live captions, the future context does not exist yet when the answer is needed. The teacher is also far too slow; the distillation paper's 1350% latency gap between 170M and 5M models illustrates the cost. We therefore evaluate the student strictly under streaming conditions and report the teacher's offline accuracy only as a reference.

Why not just adapt the first-pass recognizer federatedly? That raises the oracle ceiling, which a rescorer cannot. It is complementary, and it may absorb some of the same gains. We run FedNST-style first-pass adaptation alone, our method alone, and both together. If the combination adds nothing over first-pass adaptation, rescorer federation is not worth its communication cost, and we would report that as a negative result.

Proposed Experiments

Gating check. Before any federation, on labelled data we measure the hindsight flip precision and the WER gap between the session teacher and the causal rescorer. Xiong et al. show that past context helps. No reported result on Lacuna was found for rescoring with future cross-utterance context. If the teacher's gain over the causal model is negligible, the project stops here.

AMI meetings. This is the primary benchmark, and FMP's 4.8% relative figure sits on it. Clients are individual speakers across meetings. Sessions are meetings. The first pass is a fixed Conformer producing 10- to 50-best lists.

LibriSpeech. Consecutive utterances within a book chapter form sessions, and speakers are clients, as in FedNST's 1,173-client simulation. RescoreBERT's 6.6%/3.4% relative gains give scale for supervised rescoring. We also use LibriSpeech test-clean and test-other as held-out general-domain sets to measure forgetting after adaptation, the degradation reported in the adapter study.

Fleurs EN-US. This serves as a short-session, out-of-domain stress test where we expect little benefit.

Metrics. We report WER, relative WER reduction over first-pass-plus-centralized-rescorer, the fraction of the supervised upper-bound gap recovered, oracle WER of the lists, uploaded bytes per round, streaming latency, and WER as a function of the user-level privacy budget ε.

Decisive outcome. Two results together would settle the claim. First, hindsight distillation beats the matched re-implementation of marginal personalization and the causal self-training ablation on AMI under identical lists and budgets. Second, it does so without degrading the LibriSpeech held-out sets, at a user-level privacy budget where FMP-style noise has comparable cost. A gain that appears only with the teacher refreshed, or only without privacy noise, would narrow the claim accordingly.