Investigating Speech Recognition for Improving Predictive AAC

Jiban Adhikary, Robbie Watling, Crystal Fletcher, Alex Stanage, Keith VertanenNAACL 2019

TL;DR

This paper explores whether transcribing a conversational partner's speech can improve predictive text suggestions for Augmentative and Alternative Communication (AAC) users. They found that even with imperfect speech recognition (7-16% word error rate), language models using this transcribed speech performed nearly as well as models using perfect reference transcripts, suggesting a promising avenue for more context-aware AAC.

What Problem Does This Paper Address?

People who rely on Augmentative and Alternative Communication (AAC) devices often struggle with slow text entry rates, making real-time conversations challenging. Predictive text systems aim to speed this up by suggesting likely words or phrases. However, these systems typically only consider the text entered by the AAC user, ignoring the rich contextual information available in the partner's speech. This paper investigates if incorporating the partner's spoken words, even if imperfectly transcribed by speech recognition, can significantly enhance the accuracy of these predictive models.

Methods

The researchers explored two main areas:

  1. Speech Data Collection and Recognition: They designed an application to record audio from three different microphones (headset, laptop, conference) simultaneously while participants engaged in dialogues. They then used two commercial speech recognition engines, Google Cloud Speech-to-Text and IBM Watson Speech-to-Text, to transcribe this audio. They evaluated the Word Error Rate (WER) of these transcriptions.
  2. Language Modeling Experiments: They trained various language models (N-gram and Recurrent Neural Networks - RNNLMs) to predict the AAC user's next utterance. The core experiment involved comparing models trained on:
  3. Reference transcripts (ground truth).
  4. Speech recognition results from the partner's speech.
  5. Combinations of these and other contextual data.

They specifically investigated the impact of different microphone placements and the robustness of their models to injected noise.

Datasets

  • Amazon Turk Dialogues: The initial text dialogues were sourced from Vertanen (2017), comprising 1,419 dialogues. After filtering offensive content, 1,154 dialogues remained.
  • Collected Audio Data: 196 dialogues (1,176 utterances) were collected by recruiting 14 participants who recorded their turns in the dialogues using the custom application.
  • Movie Dialogues: For training two-sided dialogue language models, a dataset of 83K dialogues (305K turns, 3.2M words) from movies was used.
  • Twitter Data: A large dataset of 50M words from Twitter was used for training and filtering language models.

Key Insights and Ideas

  • Speech Recognition is "Good Enough": Even with WERs of 7-16%, speech recognition results were sufficient to train language models that performed nearly as well as those using perfect transcripts. This is a crucial finding, as perfect transcription is often unattainable in real-world scenarios.
  • Context is King (Even Imperfect Context): The two-way nature of conversation provides valuable context. Incorporating the partner's speech, even with errors, significantly aids prediction.
  • Microphone Choice Matters: While headset and laptop microphones yielded lower WERs, the conference microphone, placed further away, showed a sharper decline in performance with increasing noise. This highlights the practical trade-offs in microphone deployment.
  • RNNLMs Adapt Well: Recurrent Neural Networks, particularly GRUs, showed strong performance, adapting well to dialogue content even when trained on general text like Twitter data. They outperformed N-gram models in capturing long-range dependencies.
  • Ensemble Models Shine: Combining various models (N-gram, RNNLM, cache models) and data sources (Twitter, dialogues) led to the best performance, demonstrating the power of a multi-faceted approach.

Key Results

  • WER: Google's speech recognizer achieved lower WERs (7.0-8.9%) compared to IBM Watson (10.5-16.0%) across different microphones in quiet environments.
  • Noise Impact: Even with injected street noise at a moderate level (0.3), Google and IBM's WERs on headset/laptop microphones remained below 40%. The conference microphone's performance degraded more significantly with noise.
  • Language Model Performance: The best ensemble model, incorporating Twitter data, movie dialogues, and a unigram cache, achieved a perplexity of 66.3 on the test set. When using speech recognition results instead of reference transcripts, the perplexity only increased slightly to 66.6, even with a conference microphone (8.9% WER). This indicates minimal loss in predictive accuracy due to speech recognition errors.

Limitations and Future Work

  • Quiet Environment Assumption: The collected audio data was recorded in a quiet office. The paper acknowledges that real-world noisy environments could present greater challenges.
  • Participant Fluency: Participants were given verbatim text, potentially leading to more fluent speech than spontaneous conversation.
  • Privacy Concerns: The use of cloud-based speech recognition raises privacy implications. Future work should explore on-device recognition.
  • Practical Improvement: While perplexity gains are shown, the paper emphasizes the need for further work to demonstrate actual improvements in AAC user keystroke savings and end-user performance.
  • Hyperparameter Tuning: The authors suggest that more extensive hyperparameter tuning for RNNLMs could yield further improvements.
  • Scaling: Investigating how these methods scale to substantially larger datasets is also a future direction.

Summary updated

BibTeX
@inproceedings{adhikary2019investigating,
  author = {Jiban Adhikary and Robbie Watling and Crystal Fletcher and Alex Stanage and Keith Vertanen},
  title = {{Investigating Speech Recognition for Improving Predictive AAC}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure