TL;DR
State-of-the-art Automatic Speech Recognition (ASR) models are typically trained to smooth over speech disfluencies like "uhm" or word repetitions, which can lead to information loss or text hallucinations. This paper applies continual learning (CL) techniques to integrate explicit disfluency tokens into a pretrained Whisper model without destroying its general speech recognition capabilities. In doing so, the authors uncover a universal, specialized cross-attention head mechanism that the model naturally develops to track these hesitations.
What Problem Does This Paper Address?
When we talk, we constantly hesitate, repeat words, cough, or pause. While humans navigate these disfluencies seamlessly, standard ASR systems are heavily optimized to output "clean" text, completely omitting these events.
However, verbatim transcription is critical for several downstream applications. In clinical settings, tracking hesitations and pauses can help detect cognitive decline, dementia, or speech impairments. In standard applications, omitting them can cause alignment issues or lead the model to hallucinate words to fill the gaps.
The challenge lies in adapting a massive, pretrained model to recognize these subtle disfluency markers using small, specialized datasets. A naive fine-tuning approach causes catastrophic forgetting—the model learns to spot "uhm" but loses its general-domain language capabilities. Jointly retraining the model on everything is computationally expensive and often impossible due to data privacy constraints in clinical datasets.
Methods
The authors address this by framing disfluency detection as a Continual Learning (CL) problem. They group disfluencies into four explicit token types:
FILLER(e.g., "uh", "uhm")REP(word- and phoneme-level repetitions/revisions)DISRUPT(coughs, laughs)PAUSE
Using whisper-small.en as a backbone, they evaluate four distinct domain-incremental CL strategies to introduce these tokens sequentially:
- Elastic Weight Consolidation (EWC): A regularization method that penalizes changes to parameters critical to previous tasks.
- Experience Replay (ER): Retains a small rehearsal buffer of old data to mix into new training batches.
- A-GEM: A gradient-based approach that ensures updates don't conflict with the gradients of the rehearsal buffer.
- Weight Averaging (WA): A rehearsal-free method that averages the weights of the base model and the newly fine-tuned model.
To understand how the model adapts internally, they utilize a head-masking explainability method to calculate token-level head importance ("top-10 lift") and perform zero-masking ablations to verify if specific attention heads specialize in disfluencies.
Datasets
The study leverages three datasets from the TalkBank repository, all sharing the unified CHAT transcription format:
- Standard Malaysian English (SME) Corpus: Healthy L2 English speakers (11.79 hours), primarily containing fillers and repetitions.
- Pitt Corpus: Elderly speakers with dementia or Alzheimer's disease (12.11 hours used).
- Delaware Corpus: Speakers with mild cognitive impairment (9.72 hours).
- LibriSpeech (test-clean): Used as a baseline benchmark for clean, unperturbed speech to evaluate catastrophic forgetting.
The distribution of disfluencies across these datasets varies significantly, as illustrated below:
Key Insights and Ideas
- The Plasticity-Stability Trade-off: There is a stark conflict between general ASR performance and marker learning. Methods like Weight Averaging (WA) excel at preserving general word error rates (pWER) but fail completely at emitting new disfluency tokens. Conversely, Experience Replay (ER) and A-GEM successfully learn the markers but suffer a slight penalty in general ASR accuracy.
- A Shared Internal Mechanism: Intriguingly, regardless of which CL method successfully managed to emit the disfluency tokens, the model always relied on the exact same small set of decoder cross-attention heads to do it. Marker learning isn't method-dependent; it triggers a universal structural specialization within the Transformer architecture.
Key Results
- Token Introduction: In the initial phase (training on SME), Experience Replay (ER) and A-GEM achieved strong marker micro-F1 scores (~0.73–0.75), while Weight Averaging failed to output markers entirely (0.00 F1) despite keeping LibriSpeech error rates low.
- Sequential Adaptation: When moving sequentially through the more difficult datasets (Pitt and Delaware), Experience Replay (ER) emerged as the best choice for marker retention, achieving the highest aggregate macro-F1 score of 0.49 and showing virtually zero forgetting (Forgetting Measure = 0.02).
- Ablation Proof: To prove that the identified attention heads were truly responsible for disfluencies, the authors zeroed them out. Masking just the top 5 heads reduced
FILLERtoken emissions by 57% while leaving the underlying text accuracy (pWER) virtually untouched (only a +0.42% error increase), confirming highly localized circuit specialization.
Limitations and Future Work
The authors note that their study was restricted to a single backbone architecture (whisper-small.en) and a single sequential task ordering (SME $\rightarrow$ Pitt $\rightarrow$ Delaware).
Because a clear trade-off persists—where the best model for text accuracy (WA) is the worst at marker retention, and the best for marker retention (ER) compromises text accuracy—future work must focus on architectural or hybrid CL methods designed specifically to bridge this gap.
Summary updated