A new family of real-time noise-removal systems does not filter the noisy signal. Instead, it predicts the clean speech as a sequence of discrete codes from a neural audio codec (a compressor that turns audio into a small vocabulary of symbols and back), and then decodes those codes into a waveform. This works well for typical voices. Our claim is that it has a predictable blind spot inherited from text tokenizers. A rare word typed into a language model breaks into many unfamiliar subword pieces, and the model's predictions degrade. In the same way, a speaker whose voice the codec rarely saw (a child, a heavily accented talker, a dysarthric speaker) produces rare, unstable code sequences. The code-predicting model then "corrects" them toward typical voices, and identity and atypical phonetics are lost.
We propose to measure this speaker-induced token shattering and to fix it where it is cheapest. The fix is a tiny speaker-specific adapter in the codec's decoder (the "detokenizer"), fitted from 3–10 seconds of enrollment audio. The code-predicting model stays frozen and speaker-independent, and the adapter adds no runtime latency. In two sentences: token-based enhancement fails on unusual voices because those voices fall into rare regions of the code vocabulary. Rather than retraining the predictor or bolting on a speaker-embedding network, we let the predictor emit the codes it knows and teach a few hundred decoder parameters to turn them back into this particular person's voice.
Core Claim
Our starting point is Low-latency Speech Enhancement via Speech Token Generation. That work trains a 4.5M-parameter causal autoregressive Transformer to predict TF-Codec tokens (16 codebooks at 50 Hz) frame by frame. It uses an explicit alignment that binds each noisy frame to the previous clean token. On the DNS blind set it beats the masking baseline TFNet on noise suppression and intelligibility. The model is entirely speaker-independent, and the paper does not analyze how its token vocabulary behaves on unseen or atypical voices. It also shows how brittle such generators are: replacing explicit alignment with prefix conditioning pushes word error rate above 60%.
The tokenization literature gives us a mechanism to suspect. Heavier fragmentation on rare vocabulary tracks sharp rises in perplexity (Alan Ritter et al.). In speech, StableToken showed that small input changes push frame embeddings across quantization boundaries, so the code sequence changes. StableToken studied background noise. We hypothesize the same boundary jitter is driven by speaker atypicality.
We make three specific claims, each falsifiable:
- Rare codes on clean speech. For atypical speakers, the clean-speech codes are measurably rarer and less stable. This shows up as higher perplexity under the frozen generator and higher disagreement between generated and reference codes.
- Errors concentrate in identity and atypical phonetics. Those errors fall disproportionately on speaker similarity and word error rate for those speakers, not on background suppression.
- The decoder is the right place to adapt. A conditional scale-and-shift adapter in the codec decoder, fitted from seconds of audio, recovers most of the loss. It does so better than adapting the generator itself, at equal parameter count and with zero added algorithmic latency.
Research Program
Diagnostic stage. We first build the diagnostic. We reproduce the causal token-generation system: TF-Codec if obtainable, otherwise a causal 50 Hz residual-quantized codec with the same codebook layout and explicit alignment. We train it speaker-independently on DNS Challenge data. We then define two speaker-level fragility measures:
- the frozen generator's per-frame perplexity on teacher-forced clean codes of a held-out speaker;
- a unit edit distance, following StableToken, between codes the generator emits from noisy input and codes the codec assigns to the clean reference.
We plot both against a speaker-typicality score, meaning distance from the training-speaker distribution in an evaluation-only speaker-verification space. We also plot them against downstream speaker similarity and word error rate. If perplexity and edit distance do not rise with atypicality, claim 1 fails, and the project reduces to ordinary personalization. That outcome is itself worth reporting, given the open question in the file about how acoustic token shattering behaves under real-time corruption.
Adaptation method. Let $x$ be the clean enrollment clip (3–10 s) and $n$ a randomly drawn noise. Let $\hat{c} = G(x+n)$ be the code sequence the frozen generator $G$ emits on the noise-augmented enrollment, and let $D_\phi$ be the codec decoder whose normalization layers receive a speaker-specific scale and shift $\phi$. We fit only $\phi$:
$$\phi^* = \arg\min_\phi \; \mathbb{E}_{n}\big[\, \mathcal{L}_{\text{rec}}\big(D_\phi(\hat{c}),\, x\big) \big] + \lambda \,\lVert \phi - \phi_0 \rVert^2,$$
where $\mathcal{L}_{\text{rec}}$ is the codec's own multi-resolution spectral reconstruction loss, $\phi_0$ is the identity (speaker-independent) setting, and $\lambda$ controls how far the adapter may drift.
The design borrows from three sources. The conditional layer-normalization idea comes from Mingjian Chen et al., who adapt text-to-speech from one minute of speech. The "update very few, highly leveraged parameters" principle comes from Peter Bell, who adapts ASR to children by updating 80 filter cut-offs. The pull toward the speaker-independent solution plays the role that KL regularization plays in KL-Divergence Regularized DNN Adaptation.
The key difference from all three is what the adapter learns. Because it is trained on the generator's actual, typicalized outputs, it learns the mapping "codes this generator produces for this speaker → this speaker's real voice." In effect, it is a personal detokenizer for the shattered vocabulary. A scale-and-shift on normalization layers can be folded into the decoder's weights after enrollment, so the streaming path is byte-for-byte as expensive as before.
Speaker-normalizing encoder transform (secondary arm). A second arm tests whether the encoder side matters. Here a matching affine transform before quantization is fitted to make the speaker's clean codes less rare under the frozen generator, trained with a straight-through gradient. This transform only changes which codes become adaptation targets, not the inference path. We expect it to help mainly for the most atypical speakers, such as children's speech of the kind studied in Children's Speech Recognition through Discrete Token Enhancement. That work discretized SSL features for offline children's ASR but never touched streaming enhancement.
Noisy enrollment. Enrollment noise is the known weak point of few-second personalization (Personalized SE Without a Separate Speaker Embedding Model). We will test it directly. When the enrollment clip itself is noisy, the target $x$ is replaced by the speaker-independent system's own enhanced enrollment. We will report how much of the gain survives this circular target.
Baselines
Naive reference. The do-nothing reference is the frozen speaker-independent token generator from Low-latency Speech Enhancement via Speech Token Generation, run with no enrollment. On the DNS blind set that paper reports SIG 4.27, BAK 4.63 and WER 17.19%, against TFNet's 4.06, 4.41 and 24.82%. Our reproduction must land near these numbers before any adaptation result is meaningful. Beating this reference means improving speaker similarity and target-speaker WER on the personalized sets without lowering BAK.
Strongest continuous personalized systems. The strongest external competitors are two continuous-domain personalized enhancers.
The first is Cross-Seq+ from Real-Time Personalised SE Transformers with Dynamic Cross-attended Speaker Representations. It cross-attends frame by frame to enrollment hidden states from a 3-second clip, with 6.1M parameters. That paper reports a 1–2 dB absolute SDR gain and a 10–30% relative WER reduction over SpEXpc and AvaTr on a close-talk LibriSpeech-plus-noise set.
The second is PVQE from Personalized SE Without a Separate Speaker Embedding Model. It pools DeepVQE's internal states as the voice profile. That paper reports a real-time factor of 0.0135 for PVQE-S, more than 0.2 MOS better BAK than the DNS 2023 winner for the large model, and 2.7× fewer over-suppressed target frames than a separate-embedding approach.
These are the strongest competitors because they personalize in real time without any token bottleneck. They represent the "tokens are unnecessary" position. We will retrain both on the same DNS training data, use identical enrollment clips (3 s and 10 s), and match their parameter counts and tuning budget to ours. We will report their algorithmic latency and real-time factor alongside our own. Beating them requires matching or exceeding their target-speaker WER and speaker similarity on atypical speakers, while keeping the token system's BAK and DNSMOS advantage.
Generator-side adaptation. The ablation that isolates our key ingredient adapts the generator instead of the decoder. We fine-tune the top two Transformer layers on the same enrollment pairs, with a KL penalty toward the speaker-independent token distribution. This transplants the recipe of KL-Divergence Regularized DNN Adaptation, which at −6 dB SNR with 10 s reports PESQ 1.720 versus 1.629 and STOI 0.658 versus 0.605 on WSJ0 with NOISEX-92. We will also run a parameter-matched variant (a few hundred to a few thousand parameters, like ours) so the comparison is about location, not capacity.
Encoder-side and joint ablations. We will also run the encoder-transform-only arm and the joint arm, to separate "fix the rare codes" from "fix the decoding of rare codes."
Why not just…?
Why not just use a continuous personalized enhancer and skip tokens entirely? This is the most serious threat, and we treat it as a possible outcome rather than a strawman. The case for tokens rests on the token generator's reported advantage over the masking baseline TFNet on noise suppression and WER (Low-latency Speech Enhancement via Speech Token Generation). That comparison is not personalized, and it is not against Cross-Seq+ or PVQE. Our head-to-head on the close-talk and DNS personalized sets settles the question for typical speakers and for children separately. If a continuous system wins on both, we will report that personalizing token generators is not worth it at this latency. If the token system wins only on typical speakers, the fragility diagnosis is confirmed but the remedy is insufficient.
Why not just fine-tune the generator with KL regularization, the proven low-resource recipe? The recipe is proven on feed-forward spectral mappers, which have no discrete prior. In a token generator, the failure we hypothesize is that the prior itself pulls rare speakers toward typical codes. Nudging the top layers fights that prior with 10 seconds of data, and risks the brittleness the alignment ablation exposed. Adapting the decoder leaves the prior intact. The parameter-matched generator-side baseline tests this directly. If it equals ours, the "detokenizer" framing adds nothing.
Why not just condition the generator on enrollment codes as a prompt, the way Cross-Seq+ attends to enrollment states? This avoids any fitting step, which is appealing. However, it changes the generator's input structure, and the only evidence in the file on loosening alignment in this model family is the jump to over 60% WER under prefix conditioning. Cross-attention to an enrollment sequence is gentler than a prefix, so we include it as an extra baseline trained from scratch with the same data. We do not assume it fails.
Why not just make the tokenizer noise-robust, with StableToken-style voting or noise augmentation? Voting attacks noise-induced boundary jitter. StableToken cut unit edit distance from 26.17% to 10.17%, but on a non-causal Whisper-large-v3 backbone. Our hypothesis concerns speaker-induced jitter on clean speech, which noise robustness does not address. The diagnostic stage tests this by measuring fragility on clean codes. If speaker atypicality adds no fragility beyond noise, this alternative is the right one, and we would redirect toward a causal, lightweight voting quantizer.
Why not just run an offline generative restorer like Genhancer? Genhancer reaches Q-MOS 4.03 on DAPS Real with non-causal MaskGIT decoding and Whisper features. That design cannot meet the DNS real-time requirement of at most 40 ms, so it is out of scope as a competitor.
Proposed Experiments
Diagnostic check. On DNS-noised LibriSpeech speakers, WSJ0's eight adaptation speakers with NOISEX-92 noise, and children's speech from MyST and CMU Kids mixed with DNS noise, we measure the correlation between speaker typicality and the two fragility measures. We then test whether fragility predicts the per-speaker drop in speaker similarity and WER. No reported result was found on Lacuna for token-based enhancement on children's speech, so these numbers will be new.
Main comparison. We evaluate adaptation at 3 s and 10 s of enrollment on:
- the DNS personalized evaluation used by PVQE (DNSMOS SIG/BAK/OVRL and target-speaker over-suppression);
- the close-talk personalized set (SDR and WER), for the comparison to Cross-Seq+;
- WSJ0 with NOISEX-92 at −6 to 6 dB (PESQ and STOI), for the comparison to the KL recipe;
- the children's sets (WER and speaker similarity).
We report algorithmic latency, real-time factor, and parameter count for every system. This confirms that the folded adapter leaves the streaming path unchanged.
Decisive test. The result that would establish the proposal: decoder-side adaptation beats parameter-matched generator-side adaptation on target-speaker WER and speaker similarity. The advantage should grow with speaker atypicality, and BAK should stay within noise of the speaker-independent system.
Failure condition. The result that would sink it: the advantage is flat across typicality, or the continuous personalized baselines match it on children's speech.
The noisy-enrollment stress test, using self-enhanced targets, decides whether the method is usable in practice or only with clean enrollment.
References
- Low-Latency Speech Enhancement via Speech Token Generation
- NEO-BENCH: Evaluating Robustness of Large Language Models with Neologisms
- StableToken: A Noise-Robust Semantic Speech Tokenizer for Resilient SpeechLLMs
- AdaSpeech: Adaptive Text to Speech for Custom Voice
- Acoustic Model Adaptation from Raw Waveforms with SincNet
- KL-Divergence Regularized Deep Neural Network Adaptation for Low-Resource Speaker-Dependent Speech Enhancement
- Children's Speech Recognition through Discrete Token Enhancement
- Personalized Speech Enhancement Without a Separate Speaker Embedding Model
- Real-Time Personalised Speech Enhancement Transformers with Dynamic Cross-attended Speaker Representations
- Genhancer: High-Fidelity Speech Enhancement via Generative Modeling on Discrete Codec Tokens