Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR

Henri-Leon Kordt, Theresa Pekarek Rosin, Jae Hee Lee, Stefan WermterarXiv 2026 · arXiv

TL;DR

State-of-the-art Automatic Speech Recognition (ASR) models are typically trained to smooth over speech disfluencies like "uhm" or word repetitions, which can lead to information loss or text hallucinations. This paper applies continual learning (CL) techniques to integrate explicit disfluency tokens into a pretrained Whisper model without destroying its general speech recognition capabilities. In doing so, the authors uncover a universal, specialized cross-attention head mechanism that the model naturally develops to track these hesitations.

What Problem Does This Paper Address?

When we talk, we constantly hesitate, repeat words, cough, or pause. While humans navigate these disfluencies seamlessly, standard ASR systems are heavily optimized to output "clean" text, completely omitting these events.

However, verbatim transcription is critical for several downstream applications. In clinical settings, tracking hesitations and pauses can help detect cognitive decline, dementia, or speech impairments. In standard applications, omitting them can cause alignment issues or lead the model to hallucinate words to fill the gaps.

The challenge lies in adapting a massive, pretrained model to recognize these subtle disfluency markers using small, specialized datasets. A naive fine-tuning approach causes catastrophic forgetting—the model learns to spot "uhm" but loses its general-domain language capabilities. Jointly retraining the model on everything is computationally expensive and often impossible due to data privacy constraints in clinical datasets.

Methods

The authors address this by framing disfluency detection as a Continual Learning (CL) problem. They group disfluencies into four explicit token types:

  • FILLER (e.g., "uh", "uhm")
  • REP (word- and phoneme-level repetitions/revisions)
  • DISRUPT (coughs, laughs)
  • PAUSE

Using whisper-small.en as a backbone, they evaluate four distinct domain-incremental CL strategies to introduce these tokens sequentially:

  1. Elastic Weight Consolidation (EWC): A regularization method that penalizes changes to parameters critical to previous tasks.
  2. Experience Replay (ER): Retains a small rehearsal buffer of old data to mix into new training batches.
  3. A-GEM: A gradient-based approach that ensures updates don't conflict with the gradients of the rehearsal buffer.
  4. Weight Averaging (WA): A rehearsal-free method that averages the weights of the base model and the newly fine-tuned model.

To understand how the model adapts internally, they utilize a head-masking explainability method to calculate token-level head importance ("top-10 lift") and perform zero-masking ablations to verify if specific attention heads specialize in disfluencies.

Datasets

The study leverages three datasets from the TalkBank repository, all sharing the unified CHAT transcription format:

  • Standard Malaysian English (SME) Corpus: Healthy L2 English speakers (11.79 hours), primarily containing fillers and repetitions.
  • Pitt Corpus: Elderly speakers with dementia or Alzheimer's disease (12.11 hours used).
  • Delaware Corpus: Speakers with mild cognitive impairment (9.72 hours).
  • LibriSpeech (test-clean): Used as a baseline benchmark for clean, unperturbed speech to evaluate catastrophic forgetting.

The distribution of disfluencies across these datasets varies significantly, as illustrated below:

Figure 1: The distribution of disfluencies across the selected datasets. Percentages are calculated with respect to each dataset’s distribution.

Key Insights and Ideas

  • The Plasticity-Stability Trade-off: There is a stark conflict between general ASR performance and marker learning. Methods like Weight Averaging (WA) excel at preserving general word error rates (pWER) but fail completely at emitting new disfluency tokens. Conversely, Experience Replay (ER) and A-GEM successfully learn the markers but suffer a slight penalty in general ASR accuracy.
  • A Shared Internal Mechanism: Intriguingly, regardless of which CL method successfully managed to emit the disfluency tokens, the model always relied on the exact same small set of decoder cross-attention heads to do it. Marker learning isn't method-dependent; it triggers a universal structural specialization within the Transformer architecture.
Figure 2: Top-10 lift scores for FILLER (left) and REP (right) across decoder cross-attention heads computed on the train split averaged across all token emitting methods.

Key Results

  • Token Introduction: In the initial phase (training on SME), Experience Replay (ER) and A-GEM achieved strong marker micro-F1 scores (~0.73–0.75), while Weight Averaging failed to output markers entirely (0.00 F1) despite keeping LibriSpeech error rates low.
  • Sequential Adaptation: When moving sequentially through the more difficult datasets (Pitt and Delaware), Experience Replay (ER) emerged as the best choice for marker retention, achieving the highest aggregate macro-F1 score of 0.49 and showing virtually zero forgetting (Forgetting Measure = 0.02).
  • Ablation Proof: To prove that the identified attention heads were truly responsible for disfluencies, the authors zeroed them out. Masking just the top 5 heads reduced FILLER token emissions by 57% while leaving the underlying text accuracy (pWER) virtually untouched (only a +0.42% error increase), confirming highly localized circuit specialization.

Limitations and Future Work

The authors note that their study was restricted to a single backbone architecture (whisper-small.en) and a single sequential task ordering (SME $\rightarrow$ Pitt $\rightarrow$ Delaware).

Because a clear trade-off persists—where the best model for text accuracy (WA) is the worst at marker retention, and the best for marker retention (ER) compromises text accuracy—future work must focus on architectural or hybrid CL methods designed specifically to bridge this gap.

Summary updated

Abstract

Despite advances in large-scale Automatic Speech Recognition (ASR), disfluent speech remains challenging, as state-of-the-art systems are often optimized to omit disfluencies, leading to information loss and hallucinations. Prior work has focused on verbatim transcription and the integration of disfluency markers, but adapting models on limited datasets can lead to catastrophic forgetting of general-domain knowledge. We address this gap by leveraging continual learning (CL) with explicit disfluency tokens. We first introduce these tokens into a pretrained ASR model to establish stable token mechanisms, and then continue training on additional datasets with varying disfluency distributions. Through a detailed analysis of model dynamics during training, we identify a trade-off between marker learning and ASR performance, and a consistent cross-attention head mechanism shared across CL methods.
BibTeX
@misc{kordt2026learning,
  author = {Henri-Leon Kordt and Theresa Pekarek Rosin and Jae Hee Lee and Stefan Wermter},
  title = {{Learning to Hear Hesitation: Continual Learning for Disfluency-Aware ASR}},
  year = {2026},
  eprint = {2606.14391},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2606.14391},
}

Loading related content...

Figure