Learning to Correct Self-Adaptation: Federated Mapping of Noisy Speaker Estimates for Atypical Speech

Speech recognizers fail most on the speakers who need them most: older adults with cognitive decline and people with dysarthria, whose voices differ sharply from the training data. The usual fix without human transcripts is self-adaptation. The model transcribes a speaker, treats its own transcript as truth, adjusts a few speaker-specific parameters, and transcribes again. For these speakers the first transcript is badly wrong, so the model learns its own errors. Their recordings are also medical data that cannot be pooled on a server. This proposal targets that combination: adapting a speech model to individual atypical speakers on their own devices, without labels and without centralizing audio.

The core idea comes from an older line of speech research. A speaker adaptation that is crude but robust, estimated from the model's own noisy transcript, can be turned into a good adaptation by a separate mapping. That mapping is learned once, from speakers whose true transcripts are known. We propose to learn this correction map with federated learning, meaning collaborative training in which each site keeps its data and only model updates leave it. Clinics that hold transcribed patient speech train the map jointly. Unlabeled home devices then apply it to their own noisy self-estimates. The map is trained to answer one question: given what a speaker's adaptation looks like when estimated from the model's mistakes, what would it have looked like if estimated from the truth?

Core Claim

Unsupervised speaker adaptation degrades because the adaptation step absorbs first-pass errors. The decoupled design of Adaptive training using discriminative mapping transforms shows this. That paper splits adaptation into a robust per-speaker maximum-likelihood transform estimated from the first-pass transcript and a speaker-independent discriminative mapping learned offline with reference transcripts. On English conversational telephone speech (eval03), applying discriminative linear transforms directly to unsupervised data worsened word error rate (WER) from 27.0% to 28.1%. The decoupled mapping reached 25.3%. Our claim is that this decoupling transfers to modern foundation models and to the federated setting, with three changes.

First, the per-speaker estimate becomes a soft prompt for Whisper: a short learned vector sequence prepended to the encoder (acoustic) and decoder (linguistic) inputs, as in MOPSA. Second, the correction mapping is a small nonlinear network in prompt space. Third, that network is trained federatedly across labeled clinical sites from paired estimates, each pair consisting of a prompt fit to the model's own hypotheses and a prompt fit to the reference transcript. The prediction is specific. The gain over direct self-adaptation should grow as a speaker's first-pass error rate rises, because that is where self-estimates diverge most from oracle estimates.

Research Program

The base model is Whisper with separate encoder-side and decoder-side prompts, following MOPSA's split. Each speaker is one client. Clients come in two kinds: a minority of labeled clients (clinic recordings with transcripts) and a majority of unlabeled clients (deployment audio only).

On every client we compute a self-estimate. The frozen model decodes the speaker's accumulated audio. Hypotheses are filtered by confidence, and a prompt $\hat p_k$ for speaker $k$ is fit for a small fixed number of steps to the model's own hypotheses. This plays the role of the robust maximum-likelihood transform in the decoupled design. On labeled clients we also fit an oracle prompt $p^*_k$ to the reference transcripts, using the same budget.

A mapping network $M_\phi$ with parameters $\phi$ is then trained with federated averaging across labeled clients only. Each client contributes gradients of

$$ \mathcal{L}_k(\phi) = \ell_{\text{ASR}}\big(x_k, y_k; M_\phi(\hat p_k)\big) + \lambda \,\lVert M_\phi(\hat p_k) - p^*_k \rVert^2 , $$

where $x_k, y_k$ are the client's audio and reference transcripts, $\ell_{\text{ASR}}$ is Whisper's token cross-entropy, and $\lambda$ weights a regression term that pulls the mapped prompt toward the oracle. Several self-estimates per labeled client are drawn from random subsets of the audio, so the map sees how self-estimate noise varies with the amount of data. No audio or transcripts leave a client. A site shares only gradients of $\phi$, a network in a space of a few thousand dimensions. Following Federated Marginal Personalization for ASR Rescoring, which added Laplace differential-privacy noise to aggregated unigram statistics and reported under 1% WER degradation for $\epsilon \ge 0.5$, we will add noise to the aggregated map updates and report the utility curve.

An unlabeled client fits its self-estimate $\hat p_k$ once and decodes with $M_\phi(\hat p_k)$ from then on. The two-pass cost is therefore paid once per speaker at enrollment, not on every utterance. Federated Acoustic Modeling For Automatic Speech Recognition left exactly that per-utterance latency open. We will report real-time factor against the zero-shot router of MOPSA, whose advantage is a single pass with no enrollment.

Each cited line of work plays a distinct role. The decoupled discriminative mapping paper supplies the mechanism and the negative result that rules out direct discriminative self-adaptation. MOPSA supplies the prompt parameterization and the clinical benchmarks. It is fully centralized: it needs every speaker's prompt on one server for K-means clustering and router training. CAFT (Federated Acoustic Modeling For Automatic Speech Recognition) motivates the federated angle. Per-client affine transforms of log-Mel features, learned with federated pre-training, cut average WER from 16.33% to 15.13% across five silos. That transform is linear, estimated from first-pass hypotheses without any error correction, and was tested on five large silos rather than many small speaker clients.

FLowDUP is the closest structural competitor. A federated hypernetwork maps an unlabeled batch directly to personalized weights, anchored by a fraction of labeled clients. It assumes the input distribution alone carries the personalization signal, and it has only been tested on images. Our map instead consumes a first-pass self-estimate and is supervised by what the truth would have produced. That is the missing ingredient if the speaker signal relevant to recognition lives in the transcript-conditioned estimate rather than in raw audio statistics. FedPAT also stabilizes unseen clients with a global prior, but its prior is built from discrete class prototypes in image benchmarks and does not transfer directly to sequence outputs.

Baselines

The naive reference is unadapted Whisper decoding every speaker. MOPSA reports its gains relative to this model, so it anchors comparison with the literature.

The strongest existing method is MOPSA itself. It learns per-speaker prompts with speaker adaptive training, clusters them into prompt-experts, and routes test audio to a mixture zero-shot. It reports 0.86% absolute WER reduction over Whisper on DementiaBank Pitt and 1.47% absolute character error rate (CER) reduction on JCCOCC MoCA. We run it in two forms. Centralized MOPSA on the labeled speakers is a non-private reference that we do not need to beat; matching it within a small margin would already be a result. Federated MOPSA uses federated prompt training on labeled clients, server-side clustering of the shared prompts, a federated router, and zero-shot use on unlabeled clients. We found no reported result on Lacuna for a federated prompt-expert method on DementiaBank or JCCOCC MoCA, so we establish that number ourselves.

The second strong competitor is direct local self-adaptation. Each unlabeled client fits a prompt to its own confidence-filtered hypotheses and uses it unchanged. A variant replaces hypothesis fitting with entropy minimization plus a reset-to-source rule in the style of Continual Test-time Adaptation for End-to-end Speech Recognition on Noisy Speech (DSUTA). That paper reports 21.7% WER on CHiME-3 against 30.0% unadapted, and 35.8% against 60.6% for a collapsing baseline on its long multi-domain stream. This baseline is also the key ablation: it is our method with $M_\phi$ set to the identity.

We also run a CAFT-style federated per-client affine feature transform and a FLowDUP-style federated hypernetwork that maps a pooled encoder embedding of the unlabeled audio straight to a prompt. The latter tests whether the self-estimate input matters at all.

Fairness rules apply to every method. All methods use the same Whisper checkpoint, the same prompt length and placement, the same labeled/unlabeled split, the same number of federated rounds and local steps, and the same per-speaker enrollment audio. Hyperparameters for each method are tuned with an equal budget on held-out labeled speakers. Beating the field means lower WER (CER for Cantonese) on unlabeled held-out speakers than both federated MOPSA and direct self-adaptation, with paired per-speaker significance tests, under the same privacy noise. It also means a larger margin over direct self-adaptation in the worst-intelligibility speakers than in the best.

Why not just…?

Why not just train one global model with federated averaging and skip per-speaker adaptation? CAFT's own comparison answers this for acoustic heterogeneity: plain federated averaging reached 16.33% average WER, worse than both centralized training and client-adaptive training. A single model must also average over speakers whose atypicality pulls in different directions. We still include a federated prompt-tuned global model, and if it matches our method on the hardest speakers, per-speaker adaptation is not worth its enrollment cost.

Why not just self-adapt each speaker's prompt locally on its own transcripts? This is the real threat. With confidence filtering and resets, it may be close enough. The evidence against it is indirect: unsupervised discriminative transforms raised WER from 27.0% to 28.1% in the decoupled-mapping paper, and continual adaptation without resets collapsed to 60.6% WER in the DSUTA study. Neither tested prompt adaptation of a foundation model on atypical speech. Our identity-map ablation settles it directly. If the learned map does not beat strong self-adaptation, especially on high-error speakers, the core claim is false.

Why not just federate MOPSA's zero-shot router and avoid enrollment entirely? The router is faster and needs no per-speaker fitting. Its expressiveness is limited to mixtures of a few cluster centers learned from labeled speakers, and MOPSA itself notes that extreme heterogeneity strains global clusters. The map instead starts from a per-speaker estimate. We compare both across intelligibility bands and enrollment lengths, and we will report honestly if the router wins once enrollment audio is short.

Why not just use a FLowDUP-style hypernetwork from raw audio? It is single-pass and learned federatedly, but it assumes the audio distribution reveals the right adaptation. For speech, the speaker's lexical and articulatory deviations may only show up once a transcript is attempted. The hypernetwork baseline, given identical labeled clients and compute, tests this. A tie would mean the self-estimate adds nothing.

Proposed Experiments

The main experiments use DementiaBank Pitt (English elderly speech) and JCCOCC MoCA (Cantonese cognitive-assessment interviews), with speakers as clients. Labeled fractions are 10%, 25% and 50% of speakers, with evaluation only on speakers whose data never touched map training. Dysarthric speech in TORGO and UASpeech forms the stress test, since UASpeech's intelligibility bands give the stratification the core prediction needs. Personalized Adversarial Data Augmentation for Dysarthric and Elderly Speech Recognition reports up to 3.0% absolute WER reduction on DementiaBank and 0.91% on TORGO from centralized augmentation. Those gains are complementary, so we test whether augmentation on labeled clients improves the map.

The decisive check plots, per speaker, the WER gain of our method over the identity-map ablation against that speaker's unadapted WER. The claim predicts a positive slope. A flat or negative slope refutes the mechanism even if average WER improves. A second check measures how much of the gap between the self-estimate $\hat p_k$ and the oracle $p^*_k$ the map closes on held-out labeled speakers, and whether closing it predicts WER. Remaining checks measure robustness to fewer labeled sites (does the map generalize from 10% of speakers?), the enrollment-audio curve against the zero-shot router, and WER under differential-privacy noise on map updates across a range of $\epsilon$. We also run a membership test asking whether shared map gradients reveal which speakers participated, since prompts encode speaker identity.