Towards Detection and Remediation of Phonemic Confusion

Francois Roewer-Despres, Arnold Yeung, Ilan KoganACL 2021

TL;DR

This paper introduces a framework for detecting and mitigating phonemic confusion in dialogue systems, a crucial step for reducing communication breakdowns. They developed and evaluated two neural network models that outperform a weighted n-gram baseline in predicting phoneme confusion, demonstrating early promise for their proposed framework.

What Problem Does This Paper Address?

Effective communication is paramount for interactive AI systems, especially in high-stakes domains like healthcare. Communication breakdown, where users misunderstand the AI or vice-versa, can severely hinder system success. While previous work has focused on detecting communication breakdown after it occurs (e.g., through analyzing speech patterns), this paper proposes a proactive approach: a "confusion-mitigation framework" integrated into the AI's language generation process. This framework aims to identify and avoid utterances that are likely to cause confusion before they are spoken. As a first step, this work focuses specifically on detecting phonemic confusion – the misidentification of speech sounds by a listener.

Methods

The core of this work is the development of a "confusion prediction component" that estimates the probability of phonemic confusion for a given utterance. To achieve this, the authors propose and evaluate two neural network architectures:

  • Fixed-Context Network: This model uses a fixed window of 9 phonemes (the current phoneme and 4 surrounding it) as input. It employs embedding layers, followed by several fully-connected layers with ReLU activations, and a sigmoid output for predicting error probability.
  • Residual Bidirectional LSTM Network: This architecture processes the entire reference transcription. It uses embedding layers, a bidirectional LSTM layer, and residual linear blocks with ReLU activations, culminating in a sigmoid output.

Both models are trained to predict the probability of a phoneme being mis-transcribed. They also introduce a weighted n-gram baseline for comparison, which considers past phonemes in a sequence. A key methodological aspect is the handling of class imbalance in their dataset, where transcription errors are significantly rarer than correct transcriptions. They address this by using a weighted binary cross-entropy loss function, giving 10 times more weight to transcription errors.

Datasets

The authors introduce a new dataset called the Parallel Audiobook Corpus. This corpus consists of 121 hours of recorded speech from audiobooks, aligned with their text transcriptions. For their preliminary work, they used an Automatic Speech Recognition (ASR) system (Google Cloud's) as a proxy for human transcribers to generate "hypothesis transcriptions" from the audio. The original text transcriptions serve as "reference transcriptions." This dataset contains 84,253 parallel transcriptions, split into training, validation, and test sets.

Key Insights and Ideas

  • Proactive Confusion Mitigation: The paper shifts the focus from reactive detection of communication breakdown to proactive avoidance by predicting and preventing confusing utterances.
  • Phonemic Confusion as a Key Component: Targeting phonemic confusion is identified as a tractable and important first step towards a broader confusion-mitigation framework.
  • Neural Networks Outperform Baselines: Both proposed neural architectures significantly outperform a weighted n-gram baseline, suggesting that more complex models can better capture the nuances of phonemic confusion.
  • Bidirectionality Matters: The bidirectional nature of the LSTM model, allowing it to consider both past and future phonemes, is hypothesized to be a key advantage over the unidirectional n-gram baseline.
  • Context is Crucial: The success of both neural models highlights the importance of considering surrounding phonemes (context) when predicting confusion, something traditional n-gram models handle less effectively.
  • Archaism and Vocabulary as Sources of Noise: Qualitative analysis suggests that archaic language and unfamiliar vocabulary (semantic noise) can contribute to phonemic confusion, even when the underlying sounds are distinct.

Key Results

  • Both the fixed-context network and the bidirectional LSTM network outperformed the weighted 3-gram baseline in predicting phoneme confusion.
  • The fixed-context network showed a slight performance edge over the LSTM model.
  • All models performed significantly better than random chance, demonstrating the feasibility of predicting phonemic confusion.
  • Qualitative analysis of high-confidence predictions revealed that errors often involved stop words, homonyms, near-homophones, and archaic language, suggesting the models are picking up on relevant patterns.

Limitations and Future Work

  • Proxy Data: The use of an ASR system as a proxy for human transcribers is a limitation. Future work should involve a dataset with human-annotated transcriptions for more accurate results.
  • Data Size: The current dataset is relatively small. A larger, higher-quality dataset could enable the use of more advanced architectures like transformers.
  • Limited Scope: Phonemic confusion is only one aspect of communication breakdown. Future work should address other types of confusion (e.g., semantic, pragmatic) and integrate the full confusion-mitigation framework, including generating alternative utterances.
  • Listener Modeling: The framework could be further enhanced by conditioning on specific listener priors or characteristics for personalized confusion mitigation.
  • Natural vs. Semantically Unpredictable Sentences: Investigating human phonemic confusion in different types of sentences is suggested.

Summary updated

BibTeX
@inproceedings{roewerdespres2021towards,
  author = {Francois Roewer-Despres and Arnold Yeung and Ilan Kogan},
  title = {{Towards Detection and Remediation of Phonemic Confusion}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure