How You Say It Matters: Measuring the Impact of Verbal Disfluency Tags on Automated Dementia Detection

Shahla Farzana, Ashwin Deshpande, Natalie PardeACL 2022

TL;DR

This paper investigates how verbal disfluencies, often removed by standard speech-to-text systems, impact automated dementia detection. They found that removing these disfluencies, whether manually annotated or automatically detected, actually hurts dementia detection performance, suggesting that these "imperfections" in speech are valuable signals.

What Problem Does This Paper Address?

As the global population ages, detecting Alzheimer's disease (AD) and related dementias early is crucial. Speech and language changes are known early indicators of dementia. While some studies suggest verbal disfluencies (like repetitions, false starts, and filled pauses) are predictive of dementia, current automatic speech recognition (ASR) systems are designed to remove them to produce "clean" transcripts. This creates a disconnect: ASR systems aim for fluency, but disfluencies might be key to detecting cognitive decline. This paper bridges this gap by exploring the impact of these disfluencies, both gold-standard and automatically detected, on dementia detection performance.

Methods

The researchers used an off-the-shelf disfluency annotator to tag disfluencies in speech transcripts. They focused on two main types: repetitions (word and phrase) and corrections/retracings (word and phrase). To evaluate their impact, they systematically removed these disfluencies from transcripts in several ways:

  1. Gold Disfluency Removal: Manually annotated disfluencies were removed from the original transcripts.
  2. Fisher Disfluency Removal: Disfluencies predicted by their automatic annotator were removed from ASR-generated transcripts.

They then trained a BERT-based classifier on these modified transcripts to detect dementia status (AD vs. Healthy Control). They compared the performance of these different conditions against a baseline using the original, disfluency-rich transcripts.

Datasets

The primary dataset used is the ADRESS Challenge corpus. This corpus consists of spontaneous speech from picture descriptions (the Cookie Theft task) collected from participants diagnosed with Alzheimer's Disease (AD) or Healthy Controls (HC). The data includes audio recordings and manual transcriptions. The corpus is a subset of the Pitt corpus, which itself is a subset of the DementiaBank dataset.

Key Insights and Ideas

  • Disfluencies are Informative: Contrary to the goal of ASR systems to produce fluent output, this study strongly suggests that verbal disfluencies are valuable signals for detecting dementia.
  • ASR's "Cleaning" is Detrimental: Simply using ASR-generated transcripts without disfluencies significantly degrades dementia detection performance, highlighting the negative impact of removing these speech patterns.
  • Automatic Detection Matters: Even automatically detected disfluencies, when removed, negatively impact performance, though to a lesser extent than removing gold-standard annotations. This suggests that current automatic disfluency detection methods, while imperfect, capture useful information.
  • Repetition vs. Retracing: While both types of disfluencies are important, removing retracing (verbal backtracking to correct oneself) seemed to cause a slightly larger drop in accuracy than removing repetitions.

Key Results

  • Baseline Performance: The baseline condition (original transcripts with gold disfluencies) achieved the highest performance.
  • ASR Impact: Using ASR-generated transcripts (without disfluencies) led to significant performance drops: accuracy decreased by 17.7%, F1 for AD by 14%, and F1 for HC by 33% compared to the baseline.
  • Gold Disfluency Removal: Removing gold-annotated disfluencies (both repetitions and retracings) degraded classification accuracy by 5.6% and F1 score by 6% compared to the baseline.
  • Fisher Disfluency Removal: Removing automatically detected disfluencies (using their Fisher annotator) resulted in a smaller performance degradation: accuracy decreased by 3%, F1 for AD by 4%, and F1 for HC by 2% compared to the baseline.

Limitations and Future Work

Limitations: * The study acknowledges that ASR has a high error rate in real-world settings, which contributes to the performance degradation observed with ASR-generated transcripts. * The performance observed in development experiments showed large differences across folds, indicating a degree of brittleness, possibly due to using a large model (BERT) on a small dataset. This was addressed by using multiple runs of five-fold cross-validation. * The automatic disfluency annotator's accuracy for retracing is lower than for repetition, and it struggles with detecting longer repetitions.

Future Work: * The authors propose training an end-to-end ASR system directly on disfluent speech. This system would aim to generate richer, disfluency-annotated transcripts, potentially leading to better dementia detection systems. * They also suggest exploring the impact of disfluency on other conditions, such as autism, where filled pauses have been noted as less frequent in children with ASD.

Summary updated

BibTeX
@inproceedings{farzana2022how,
  author = {Shahla Farzana and Ashwin Deshpande and Natalie Parde},
  title = {{How You Say It Matters: Measuring the Impact of Verbal Disfluency Tags on Automated Dementia Detection}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure