Speech recognizers on phones, tablets and smart speakers increasingly personalize themselves. They keep a small set of trainable parameters that slowly adjust to the owner's voice, accent or speech impairment. These methods share one quiet assumption: every utterance the device hears comes from one known person. In real homes this is false. Devices are passed between family members, children borrow parents' tablets, and nobody logs in before saying "play the next song." A personalizing recognizer that cannot tell users apart will fit its "owner" parameters to a mixture of voices. It then gets worse for the owner and never helps anyone else.
We propose to measure this hazard and to build an adaptation scheme that treats user identity as a hidden variable rather than a given label. The device keeps a small bank of per-user adapters. An adapter is a lightweight add-on to a frozen recognizer that holds one person's acoustic adjustments. Each utterance updates these adapters in proportion to how confident the device is about who spoke it. Utterances that look like nobody known start a new adapter, and adapters that turn out to be the same person are merged. The central new ingredient is a two-way loop. Identity estimates decide which adapter learns, and the adapters, once trained, become evidence about identity. This matters most where standard speaker identification is weakest: acoustically similar family members, short commands, and users who never enrolled.
Core Claim
On shared devices, continual speaker adaptation with hard or assumed identity labels suffers measurable cross-user contamination. An uncertainty-weighted, enrollment-free adapter bank recovers most of the gap to an oracle that knows the true speaker.
The key hypothesis is that adapters carry identity information that a speaker embedding lacks. An embedding captures voice timbre. An adapter trained on someone's speech captures how that person pronounces words. This is the information that separates a mother from a daughter whose voices are similar but whose accents or articulation differ. If the hypothesis is right, the loop improves both transcription for each user and the identification of users. If it is wrong, the proposal reduces to careful soft gating, and the experiments below are designed to say so.
Where This Sits
The adaptation side supplies the machinery. Continual Learning for On-Device Speech Recognition Using Disentangled Conformers splits a Conformer into a frozen core and small trainable "augment" networks. It gets over 20% relative word error rate (WER) reduction on the 118-speaker LibriContinual benchmark with no forgetting on LibriSpeech, but its adaptation stream is perfectly partitioned by speaker. A Unified Speaker Adaptation Approach for ASR fine-tunes pruned weights per speaker, reporting up to 20.58% relative WER reduction and gains from a single 15-second utterance. It too presumes the target speaker is known. Use of Speech Impairment Severity for Dysarthric Speech Recognition factorizes Learning Hidden Unit Contribution (LHUC, per-neuron scaling factors learned per speaker) into separate speaker and severity transforms. We reuse this factorized form because it makes mixing several users' adapters cheap: mixing becomes a weighted sum of scaling vectors. All three works are method suppliers whose identity assumption we remove.
Three works address "who is speaking" at the recognition layer, and each leaves a gap our proposal targets. CASA-ASR: Context-Aware Speaker-Attributed ASR jointly transcribes and attributes speakers in meetings, but needs fixed pre-enrolled d-vectors and never adapts the acoustic model. Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR adapts the encoder using only a diarizer's timing mask, with no enrollment. That adaptation is deliberately memoryless across turns, which is the opposite of what longitudinal personalization needs. SGEM: Test-Time Adaptation for Automatic Speech Recognition via Sequential-Level Generalized Entropy Minimization adapts per utterance without labels by minimizing prediction entropy. It resets after each utterance and costs about 0.77 seconds of compute per second of audio.
The identification side motivates and constrains the design. Graph-based Multi-View Fusion and Local Adaptation: Mitigating Within-Household Confusability for Speaker Identification shows that voice-only identification fails within households: 8.85% identification error on acoustically hard VoxCeleb1 households. It fixes this with household-local score normalization and label propagation. The authors state that they freeze embeddings and do no acoustic adaptation, and their method is transductive rather than streaming. Speaker Identification for Household Scenarios with Self-Attention and Adversarial Training reports household equal error rates of 3.39% for known and 6.39% for unseen speakers on 1.5-second VCTK commands. That is the evidence that even good identifiers mislabel a nontrivial share of short commands. Quickest Intruder Detection for Multiple User Active Authentication tracks the minimum likelihood ratio across all enrolled household members so that handovers between known users raise no alarm. We borrow this statistic as our rule for detecting a new, unknown user.
Finally, work on shared devices motivates the problem itself. Feedback Effect in User Interaction with Intelligent Assistants: Delayed Engagement, Adaption and Drop-out notes that assistant logs cannot separate users sharing a device. Applying Behavioral Biometrics to Mobile Device Use Measurement in Children: Evaluating the Impact of Training Data Size, Proximity, and Type on Model Performance constrains any plan to lean on motion biometrics instead of voice. There, identification F1 falls from 0.91 to 0 when test data is 11 minutes away from training data, and to 0 across postures. We therefore do not route adaptation on non-voice biometrics in this proposal.
Mechanism
The recognizer core stays frozen. The device holds adapters $\theta_1,\dots,\theta_K$ for the users discovered so far, plus a shared household adapter $\theta_0$ that absorbs whatever is common to the room and microphone. For an incoming utterance $x$, the device computes a posterior $q_k(x)$ over users. The posterior combines two sources of evidence: a speaker-embedding score, normalized locally within the household as in Chen et al., and the adapter-decoding score $s_k(x)$. The adapter-decoding score is the recognizer's log-likelihood of its best hypothesis when run with adapter $k$, measured relative to the base model:
$$q_k(x) \propto \pi_k \, \exp\big(\alpha\, e_k(x) + \beta\, [s_k(x) - s_{\text{base}}(x)]\big),$$
Here $\pi_k$ is a usage prior, $e_k$ is the normalized embedding similarity, and $\alpha,\beta$ are weights tuned on held-out households. The adapter term is switched on only after an adapter has seen a minimum amount of speech, such as two minutes. The comparison uses a frozen snapshot of each adapter, which limits the obvious rich-get-richer failure in which an adapter becomes confident on everything it trained on.
Each adapter then receives a gradient step on $x$ weighted by $q_k(x)$. The shared adapter $\theta_0$ always receives a small weight. If $\max_k q_k(x)$ falls below a threshold, only $\theta_0$ is updated. If the minimum likelihood ratio against all known users crosses the change-detection threshold for several consecutive utterances, a new adapter is created. This is the case where the speech fits no known user. Two adapters whose embeddings and weight vectors converge are merged, which handles fragmentation.
At decoding, the device uses the posterior-weighted mixture of LHUC scalings. This costs one forward pass, unlike beam-search test-time adaptation. Training targets come from two regimes. The first uses oracle transcripts, which isolates the effect of attribution. The second uses the model's own pseudo-labels, which reflects realistic deployment.
Research Program
The work has three stages. The first quantifies the conflation hazard. We run DisentangledCL on LibriContinual speakers while injecting a co-user's utterances into the target's adaptation stream at contamination rates from 0 to 50%. Contamination rates are matched to the misattribution rates that SAASI-style and x-vector identifiers actually produce on the same households. We measure WER for the target, for the co-user, and on LibriSpeech. The second stage builds the adapter bank and compares it against the baselines below. The third stage tests the claim that adapters carry identity evidence. It asks whether adding $s_k$ reduces speaker identification error beyond local normalization alone on hard households, and whether this gain is concentrated in pairs with similar voices but different accents or pronunciation.
Baselines
The naive reference is the frozen recognizer with no adaptation. Every method must beat it, and its gap to the oracle defines the room available.
The oracle upper bound is DisentangledCL given true speaker labels. It is the strongest existing adaptation method in the file and reports over 20% relative WER reduction on LibriContinual (Continual Learning for On-Device Speech Recognition Using Disentangled Conformers). No per-speaker absolute WER was found on Lacuna for LibriContinual, so we reproduce the oracle ourselves. We run it with the same core, adapter size and data as our method. Our success criterion is recovering at least half of the gap between the best non-oracle baseline and this oracle on hard households.
The pooled household adapter runs DisentangledCL on all device audio as if it were one user. It is the strongest competitor because it may capture most of the gain if that gain comes from room and microphone rather than from individual speakers. To beat it, our method must reduce WER for each individual user, not only on average.
Hard identification with confidence gating assigns each utterance to one adapter using a locally normalized speaker embedding. Low-confidence utterances are discarded. It receives the same enrollment conditions as our method, including none, in which case identities come from online clustering, and the same threshold-tuning budget. It is the most direct simpler alternative.
Episodic SGEM, run from the official drumpt/SGEM codebase, represents memoryless label-free adaptation. Its reported number is a 15.6% average WER reduction across 12 domain-shift datasets. We compare against it at matched compute as well as unmatched. Self-speaker adaptation targets overlapping speech. We compare against it only in its memoryless form on single-talker turns, and we note that this is outside its design regime.
Ablations isolate the key ingredients. The first removes the adapter-likelihood term ($\beta=0$, embedding-only soft weights). The second replaces soft weighting with argmax assignment. The third disables new-user creation and merging.
Why not just…?
Why not just identify the speaker and load their adapter, skipping updates when unsure? This is the real threat. Gating trades contamination for discarded data, and on hard households, where voice-only identification errs 8.85% of the time (Graph-based Multi-View Fusion and Local Adaptation), it either admits errors or throws away much of the stream. It also needs enrolled references, which shared devices rarely have. If the gating baseline matches our method at every contamination level and household difficulty, we will report that simple gating suffices. The $\beta=0$ ablation shows whether the adapter-evidence loop adds anything beyond soft gating.
Why not just adapt one shared household model? Pooling avoids attribution entirely, and if device and room effects dominate, it may be enough. The first stage of the program settles this directly. If pooled adaptation's per-user WER matches the oracle's, the conflation hazard is not practically important, and the premise fails. We expect it to fail specifically on mixed households, such as one containing a dysarthric and a typical speaker drawn from UASpeech, because their adjustments pull in opposite directions.
Why not just run test-time adaptation per utterance and never track identity? SGEM needs no labels, but it resets after every utterance and costs about 0.77 seconds of compute per second of audio (SGEM). It cannot accumulate the minutes-to-hours of per-speaker data from which LibriContinual gains come. Its entropy objective also has no protection against mixed or misattributed input. We test it at matched compute. If it wins on short commands, a hybrid that initializes each utterance from the posterior-weighted adapter becomes the natural next step.
Why not just ask users to log in? Active prompts reach high compliance in controlled studies (Pérez et al.), but hands-free voice use and children's device sharing make per-command verification impractical. Friction also measurably reduces re-engagement with assistants (Feedback Effect in User Interaction with Intelligent Assistants). We treat login as an upper bound on attribution quality rather than as a competitor.
Proposed Experiments
Households are simulated three ways. The first uses LibriContinual speakers grouped into 2–5-person households, with "hard" households selected by embedding similarity following the construction of Chen et al. The second uses VCTK 4-speaker households following SAASI, which adds accent diversity and short utterances. The third uses mixed households combining UASpeech dysarthric and control speakers, and L2-Arctic non-native speakers with native ones. Usage streams interleave speakers with realistic turn-taking, including runs from a single user and sudden handovers.
We report four families of measures. The first is per-user WER, which counts every household member. The second is WER on base LibriSpeech, which measures forgetting. The third is speaker identification error and household equal error rate, reported next to SAASI's 3.39% and 6.39% and Chen et al.'s 8.85% voice-only figure. The fourth is new-user detection delay and false-creation rate.
Three results would be decisive. First, the hazard must exist: DisentangledCL's target-user WER must degrade monotonically with contamination at rates that real identifiers produce. Otherwise the proposal's motivation collapses. Second, on hard and mixed households, the adapter bank must close at least half the gap between the best of pooled and gated adaptation and the oracle, under both oracle-transcript and pseudo-label training. Third, the adapter-likelihood term must lower identification error beyond local normalization alone, with the gain concentrated in pairs with similar voices but different pronunciation.
A known limitation is that all households are simulated from read or elicited speech, since the file contains no real shared-device corpus with transcripts. The results bound the effect rather than confirm deployment behavior.
References
- Continual Learning for On-Device Speech Recognition using Disentangled Conformers
- A Unified Speaker Adaptation Approach for ASR
- Use of Speech Impairment Severity for Dysarthric Speech Recognition
- CASA-ASR: Context-Aware Speaker-Attributed ASR
- Speaker Targeting via Self-Speaker Adaptation for Multi-talker ASR
- SGEM: Test-Time Adaptation for Automatic Speech Recognition via Sequential-Level Generalized Entropy Minimization
- Graph-based Multi-View Fusion and Local Adaptation: Mitigating Within-Household Confusability for Speaker Identification
- Speaker Identification for Household Scenarios with Self-Attention and Adversarial Training
- Quickest Intruder Detection For Multiple User Active Authentication
- Feedback Effect in User Interaction with Intelligent Assistants: Delayed Engagement, Adaption and Drop-out
- Applying Behavioral Biometrics to Mobile Device Use Measurement in Children: Evaluating the Impact of Training Data Size, Proximity, and Type on Model Performance
- The Family Level Assessment of Screen Use–Mobile Approach: Development of an Approach to Measure Children’s Mobile Device Use