Re-Anchoring Speaker Adapters to a Moving Cohort

Speech recognizers personalized to one user tend to go stale. People change microphones, move into noisy rooms, catch colds, and change how fast they talk. Yet speaker personalization is almost always done once, from a short enrollment, and then frozen. This proposal treats personalization as a stream. The personal model should be kept close to the voices it currently sounds most like, not to the voice it sounded like on day one. Concretely, we keep a bank of small adapters, one per reference speaker. An adapter here means a low-rank weight update, a cheap add-on matrix that tweaks a frozen recognizer for one speaker. Every few minutes of use, we look up which reference speakers the user's recent audio most resembles. We then pull the user's adapter toward the average of those speakers' adapters while it keeps learning from new speech.

The bet is that this moving, similarity-based anchor handles drift better than the two standard reflexes. The first reflex down-weights old data on a fixed time schedule. Its right setting depends on how fast things change, which is unknown in advance. The second reflex resets the adapter when accuracy drops, which throws away everything learned and brings back the data scarcity that personalization was meant to fix. We add one piece of matrix algebra to keep the approach cheap. Because the adapter is a product of two thin matrices, its conditioning can be inspected and repaired from an $r \times r$ core. That core is tiny compared with the full layer, so the approach avoids the full singular value decomposition (SVD) that makes spectral repair of continual learners expensive.

Core Claim

Consider streaming speaker adaptation under acoustic drift with little recent audio per drift phase. A prior that is re-retrieved periodically from acoustically similar reference speakers will do two things better than fixed-rate exponential discounting and reset-on-drift. It will recover faster after change points, and it will forget less when old conditions return. It should do so without tuning a decay rate to each drift type. Spectral maintenance of the adapter factors, done in the factored form, is a secondary claim. We test it as a diagnostic first and drop it if the collapse it targets does not occur.

How the Pieces Relate to Prior Work

Cohort prior (the core mechanism). The mechanism comes from classical acoustic similarity matching. A cohort-UBM approach scores 5 seconds of enrollment against development-speaker models. It pools the 10 closest speakers into a replacement prior for MAP adaptation and reports relative equal error rate gains of 10.57% to 58.33% on TIMIT. BCAT-NN seeds Bayesian speech-synthesis adaptation from the nearest reference speaker, using 1–4 seconds of speech. Both select the cohort once, from clean enrollment, with a fixed cohort size and a linear scan over all speakers. We move the cohort idea into neural adapter space, re-run selection over time, and replace the linear scan with nearest-neighbor search over speaker embeddings.

Adapter parameterization. On the adaptation side, DoRA speaker personalization shows that low-rank adapters on the attention layers of a conformer transducer give about a 20% relative word error rate (WER) reduction on LibriSpeech, with roughly 139k–208k parameters. It adapts each speaker independently, in one batch session. P-ShareLoRA supplies the parameterization that makes cohort averaging meaningful. It shares one up-projection $B$ across users and keeps a per-user down-projection, which reached 74.65% preference accuracy on Reddit TL;DR. With a shared $B$, an average of per-speaker down-projections is an average of adapters, which is not true when each speaker has its own $B$. P-ShareLoRA and PAL, which learns user mixture weights over fixed prototypes, both treat user parameters as stationary. That stationarity is exactly what we relax.

Drift literature (the regime and the main foil). NS-DPO weights each sample by $\gamma^{T-t_i-1}$ and survives preference flips where plain DPO falls to near chance. The authors themselves note that the best $\gamma$ depends on an unknown variation budget. We replace the tuned decay with a data-driven anchor. Online-LoRA detects shifts from loss plateaus, merges and reinitializes its adapter, and reaches 47.06% on Split-ImageNet-S. It is our strongest "reset" competitor, although it has only been shown on vision. We call this drift "acoustic" rather than "preference" drift. The two share the structure that matters here: recent evidence should override stale evidence.

Spectral maintenance (the constraint). SingularClip shows that trailing singular values collapse under non-stationary training and that this stalls later learning. It fixes the problem with an exact SVD of every layer every $K$ steps, which it names as its main cost. Its evidence that one-sided spectral control and resetting fail constrains our design. We only consider spectral maintenance that addresses the trailing end of the spectrum.

Research Program

Reference bank. We train a frozen shared up-projection $B$ and one down-projection $A_k$ per reference speaker $k$, on the LibriSpeech training speakers. Each reference speaker is also indexed by the mean of a pretrained speaker embedding.

Streaming update. During a user's stream, every $M$ utterances we embed a sliding window of recent audio. We retrieve the $N$ nearest reference speakers, with similarity weights $\pi_k$ given by a softmax over negative embedding distance. We then minimize

$$ \mathcal{L}(A_u) = \sum_{t \in \mathcal{W}} \ell_t(B A_u) + \lambda \Big\| A_u - \sum_{k \in \mathcal{C}_T} \pi_k A_k \Big\|_F^2 , $$

where $A_u$ is the user's down-projection, $\ell_t$ is the transducer loss on recent utterance $t$ in window $\mathcal{W}$, and $\mathcal{C}_T$ is the cohort retrieved at the current time $T$. The strength $\lambda$ is the MAP-style weight carried over from cohort-UBM adaptation.

Choosing the cohort size. Instead of fixing $N$, we cut the cohort where the sorted distances show a gap. This targets the "static cohort sizing" limitation of the cohort-UBM paper.

Two supervision settings. We run a transcribed setting (upper bound) and a pseudo-label setting with confidence filtering (realistic).

Spectral maintenance. This uses a gauge freedom: $BA_u = (BR^{-1})(RA_u)$ for any invertible $r\times r$ matrix $R$. Because $B$ is shared, we act only on $A_u$. We compute the $r\times r$ Gram matrix $A_uA_u^\top$ in $O(r^2 n)$, eigendecompose it, and clip the singular values of $A_u$ to within a factor $c$ of their median. This follows SingularClip's rule applied to the adapter factor, and it is a projection that changes the function slightly. The gradient reaching $B$ and the output layer passes through $A_u x$, so an ill-conditioned $A_u$ narrows what the user adapter can express.

Checking whether collapse happens. We first log the singular spectrum of $A_u$ along every stream. If it does not become anisotropic under drift, the clipping component is removed from the claims.

Baselines

Frozen personal adapter. The naive reference is a DoRA/LoRA adapter trained once on the first 1–5 minutes of a user's stream and never updated, following DoRA speaker personalization. The reported figure there is about 20% average relative WER reduction over the base conformer transducer on LibriSpeech test-other speakers. We will first reproduce a comparable gain on static data, so that later comparisons start from a validated adapter. The base unadapted recognizer is reported alongside it.

Time-discounted streaming adaptation. This is the strongest drift competitor. It applies NS-DPO's weighting $\gamma^{T-t_i-1}$ from NS-DPO to the transducer loss over the full stream history, using the same adapter, rank, optimizer, and update cadence as our method. To be generous, we tune $\gamma$ separately for each drift type on held-out speakers, which our method does not get. Beating it means matching its per-type oracle performance with one shared setting, and beating it when $\gamma$ must be shared across drift types.

Merge-and-reinit on drift. This baseline follows Online-LoRA: plateau detection, merge, fresh adapter, and a small replay buffer, with its buffer size and detector thresholds tuned on the same held-out speakers.

Isolating ablations. Three ablations isolate our key ingredient. The first anchors to a cohort chosen once at enrollment and never re-retrieved; it tests whether the moving cohort matters. The second anchors to a random cohort of the same size; it tests whether similarity matters. The third anchors to the global mean of the bank, a PAL-like static prior; it tests whether a cohort beats a population average.

Spectral ablations. For the spectral component, we compare no maintenance, factored clipping, and full SingularClip on the effective weights. The last is used only as an accuracy reference with its wall-clock cost reported.

Fair comparison. All methods share data, adapter rank, compute per utterance, and the same hyperparameter search budget.

Why not just…?

Why not just discount old data with a well-chosen decay rate? This is the real threat. With oracle $\gamma$ for each drift type, discounting may match us on gradual drift. Its documented weakness is that the right $\gamma$ depends on an unknown drift budget, as the NS-DPO authors acknowledge. Discounting also has nothing to say about sudden switches where recent data is only a few utterances. The shared-$\gamma$ versus per-type-oracle comparison settles this. If a single $\gamma$ performs as well as re-anchoring across drift types, the core claim fails.

Why not just reset or reinitialize the adapter when accuracy drops? SingularClip reports that periodic resetting destroyed accumulated skill and failed on DMC Humanoid-Stand. A reset adapter also re-faces the few-seconds enrollment problem that cohort methods were invented for. Merge-and-reinit is a stronger variant than a plain reset. It is included as a baseline, and the recovery-time and return-to-old-condition metrics measure the cost of its resets directly.

Why not just initialize from a good cohort once, as BCAT-NN and cohort-UBM do? That is exactly the enrollment-only-cohort ablation. If re-retrieval adds nothing over a one-time cohort, then the "moving" part of the proposal is unnecessary and the contribution reduces to porting cohort priors into neural adapters. That would still be useful, but it is a smaller claim.

Why not just skip spectral maintenance, or use weight decay? SingularClip shows that controlling only the top singular value does not stop trailing collapse. However, nobody in the file has shown that collapse happens inside a small streaming adapter. The spectrum diagnostic decides this before any claim is made.

Proposed Experiments

Drift benchmark. Personal streams are built from LibriSpeech test-other speakers, the same population as the DoRA work. Each stream orders a speaker's utterances in time and imposes one of four drift schedules. The first is gradual noise increase. The second is a sudden channel switch using a codec change or simulated room reverberation. The third is speaking-rate change. The fourth is a recurring pattern in which the original condition returns. TIMIT provides a secondary check of cohort sizing at 5-second enrollment, following the cohort-UBM protocol. No reported result was found on Lacuna for any streaming, drift-scheduled ASR personalization benchmark, so every number on these streams will be new. The static DoRA reproduction is our only anchor to published figures.

Metrics. We measure WER averaged over the stream and utterances-to-recovery after each change point. Recovery means returning within 10% relative of the pre-drift WER. We also measure WER degradation on the return phase of recurring drift, and per-update wall-clock time. Retrieval latency is measured separately, for nearest-neighbor search versus a linear scan over the bank.

Decisive checks. There are four. First, re-anchoring with one shared setting must beat shared-$\gamma$ discounting and approach per-type-oracle discounting. Second, the moving cohort must beat both the enrollment-only cohort and a random cohort, most visibly after sudden switches. Third, gains must survive the pseudo-label setting. Fourth, if $A_u$ shows collapse, factored clipping must recover most of full SingularClip's benefit at a small fraction of its cost. Failing the first two checks would falsify the core claim. Failing the fourth would only remove the secondary one.