Adaptive speech recognition and dialogue management for users with speech disorders

I. Casanueva, H. Christensen, Thomas Hain, Phil D. GreenInterspeech 2014

TL;DR

This paper explores how probabilistic dialogue management (DM) can significantly improve the performance of voice-controlled environmental systems for users with speech disorders like dysarthria. By adapting both the speech recognition (ASR) and the DM models as the user interacts, the system becomes more robust and successful, especially for users with severe speech impairments.

What Problem Does This Paper Address?

People with severe physical disabilities often rely on assistive technologies, and voice control is a highly attractive interface. However, many individuals with physical disabilities also have speech disorders, such as dysarthria, which significantly degrades the accuracy of automatic speech recognition (ASR) systems. This low ASR accuracy makes current voice-controlled systems frustrating and unreliable for these users. The paper aims to bridge this gap by investigating how advanced dialogue management techniques can compensate for ASR errors and improve the overall user experience for dysarthric individuals.

Methods

The core of this research lies in combining an adaptive ASR system with a probabilistic dialogue management (DM) approach.

  • Adaptive ASR: The system uses Maximum A Posteriori (MAP) adaptation to fine-tune the ASR acoustic models for individual users. This means the ASR gets better as the user provides more speech data. The paper also investigates the impact of vocabulary size, finding that a smaller vocabulary (36 words) can improve accuracy and reduce the amount of adaptation data needed.
  • Probabilistic Dialogue Management (DM): Instead of a simple rule-based system, the authors employ a probabilistic DM, specifically framed as a Partially Observable Markov Decision Process (POMDP). This allows the DM to handle uncertainty from the ASR and adapt its strategy.
  • Hierarchical POMDPs: To manage complexity, POMDPs are applied at each non-terminal node of the environmental control system's hierarchical structure.
  • Adaptive DM: Crucially, the DM models are designed to adapt alongside the ASR. This means the DM's "policy" (how it decides to respond) changes as the ASR's performance improves with more adaptation data. This allows the system to be more cautious (asking for more confirmations) when ASR is poor and more direct when ASR is good.
  • Speaker-Dependent vs. Speaker-Independent: They compare DMs trained on data from a single user (SD) versus DMs trained on data from multiple users (SI).
  • Simulated User (SU): To evaluate their system, they created simulated users that mimic dysarthric speakers with varying intelligibility levels and ASR adaptation data amounts. This allows for controlled experiments.

Datasets

  • UASpeech: This is the primary dataset used. It's described as the largest available database of dysarthric speech, containing recordings from 15 speakers with a range of impairment levels. The data includes numbers, alphabet letters, command words, and common/uncommon words. The paper notes that UASpeech provides intelligibility measures for its speakers.

Key Insights and Ideas

  • Probabilistic DM is Key for ASR Errors: The research strongly supports the idea that probabilistic DM is essential for making spoken dialogue systems robust to ASR errors, especially when those errors are frequent due to speech disorders.
  • Adapting DM to ASR is Crucial: The most significant improvements come from DMs that adapt their behavior as the ASR performance improves. This "conservative-to-direct" policy shift is vital for user experience.
  • Early Interaction is Critical: Probabilistic DMs show the most substantial relative improvements when ASR adaptation data is small (i.e., early in the system's use). This makes the system usable from the start, even with poor ASR.
  • Speaker-Independent Models Boost Performance: Training DM models on data from multiple speakers (SI) generally outperforms models trained on a single speaker (SD), even for speaker-specific systems. This suggests that learning from a broader range of dysarthric speech patterns can lead to more generalized and effective dialogue strategies.
  • Smaller Vocabularies Help: Reducing the ASR vocabulary size can significantly improve accuracy and reduce the amount of adaptation data needed, making systems more practical for users with limited speech capabilities.

Key Results

  • Significant Improvements: The adaptive ASR + probabilistic DM approach yielded mean relative improvements of 9% in dialogue length and 25% in success rate compared to a baseline local confidence score DM.
  • Severe Dysarthria Benefits Most: For speakers with more severe dysarthria, these improvements were even more pronounced: 25% mean relative improvement in dialogue length and 75% in success rate.
  • Early Adaptation Advantage: The adaptive DM (IA) showed higher improvements, especially when ASR adaptation was minimal (poorer ASR performance).
  • SI IA Outperforms: The speaker-independent adaptive DM (SI IA) generally showed the best results across the board, particularly for low and very low intelligibility speakers.
  • Success Rate Boost: For very low intelligibility speakers, the SI IA approach achieved up to an 80% improvement in success rate.

Limitations and Future Work

  • Simulated Users: The experiments relied on simulated users. While this allows for controlled evaluation, real-world user studies are needed to confirm these findings with actual dysarthric individuals.
  • Longer Interactions: The paper notes that improved success rates for severe dysarthric speakers may come at the cost of longer dialogues, which could be tiring. Future work could focus on optimizing dialogue length while maintaining high success rates.
  • Computational Tractability: While POMDPs offer theoretical advantages, their computational complexity can be a challenge. The paper uses approximations and hierarchical structures to address this, but further research into efficient solving methods is always valuable.
  • Real-World Deployment: The authors suggest that this adaptive ASR and DM framework is particularly promising for systems like their previously developed "homeService" environmental control system, implying future work would involve integrating and testing this combined approach in real-world scenarios.

Summary updated

BibTeX
@inproceedings{casanueva2014adaptive,
  author = {I. Casanueva and H. Christensen and Thomas Hain and Phil D. Green},
  title = {{Adaptive speech recognition and dialogue management for users with speech disorders}},
  booktitle = {Interspeech},
  year = {2014},
}
Figure