Non-Native Users in the Let’s Go!! Spoken Dialogue System: Dealing with Linguistic Mismatch

Antoine Raux, Maxine EskenaziNAACL 2004

TL;DR

This paper explores the challenges of spoken dialogue systems interacting with non-native English speakers, focusing on "linguistic mismatch" beyond just accent. They introduce a method for generating adaptive confirmation prompts to help non-native users learn idiomatic expressions and improve system performance.

What Problem Does This Paper Address?

Spoken dialogue systems (like voice assistants) are becoming ubiquitous, but they often struggle when users aren't native speakers of the language. This isn't just about accents (acoustic mismatch); it's also about how non-native speakers use language differently (linguistic mismatch). This includes variations in word choice and sentence structure, which can lead to errors in speech recognition and natural language understanding. As more people rely on these systems for information, it's crucial to make them accessible to a diverse user base, including non-native speakers. The paper aims to understand and mitigate this linguistic mismatch to improve system performance and potentially aid language learning.

Methods

The researchers developed the "Let's Go!!" bus information system, a task-based dialogue system for Pittsburgh. They collected spontaneous speech data from both native and non-native English speakers.

Their core contributions are:

  • Analysis of Linguistic Mismatch: They systematically analyzed the differences in word choice (leading to Out-Of-Vocabulary words) and syntax between native and non-native speakers, and how these impact language model perplexity and grammar parsing.
  • Impact of Non-Native Data: They investigated the effect of incorporating non-native speech data into both the language model and the grammar.
  • Adaptive Lexical Entrainment: They proposed and implemented a method for generating adaptive confirmation prompts. This involves finding the closest "target prompt" (a phrase the system expects) to the user's utterance, even if the utterance is slightly misrecognized or ungrammatical. The system then emphasizes the differing words in its confirmation, aiming to guide the user towards more idiomatic expressions.

Datasets

  • "Let's Go!!" System Data: Spontaneous speech data collected from international students and native English speakers interacting with the bus information system. This includes data from June 2003 (119 calls from 11 non-native speakers, 25 calls from 4 native speakers) and August 2003 (47 calls from 6 non-native speakers). An additional corpus of 1757 spontaneous non-native utterances from unsolicited callers was also used.
  • CMU Communicator System Data: Used for training acoustic models.

Key Insights and Ideas

  • Linguistic Mismatch is Significant: Beyond accent, differences in word choice (e.g., using "intersection of A and B" vs. "A and B") and syntax (e.g., varied ways to ask "When does the bus leave?") significantly degrade system performance.
  • Non-Native Data Helps: Incorporating non-native speech data into language models and grammars demonstrably reduces OOV rates and perplexity for non-native speakers.
  • Adaptive Prompts as a Learning Tool: The adaptive confirmation prompt strategy is a promising way to bridge the gap. By echoing user input with slight corrections and emphasis, the system can subtly guide users towards more standard or idiomatic language without being overly intrusive.
  • NLU is More Robust: The Natural Language Understanding (NLU) module, designed to be flexible to handle speech recognition errors, is less sensitive to non-native linguistic variations than the language model.

Key Results

  • Word Error Rate (WER): Native LM on native data: 20.4%; Native LM on non-native data: 52.0%. This highlights the significant impact of linguistic mismatch.
  • OOV Rate: Non-native speakers had a 3.09% OOV rate compared to 1.2% for native speakers.
  • Perplexity: Non-native utterances had a perplexity of 36.55, 59.7% higher than native utterances (36.55 vs. 22.89) when using the native language model.
  • Parsing: Native speakers achieved higher parsing rates (63.3% words parsed, 56.4% fully parsed) compared to non-native speakers (56.0% words parsed, 49.7% fully parsed) when using the native grammar.
  • Mixed Model Improvement: Using a mixed language model (native + non-native data) reduced OOV rate by 56.6% and perplexity by 23.6% for the non-native test set.
  • Adaptive Prompts: The adaptive confirmation prompt method showed success in repairing utterances that would otherwise not be parsed, especially when recognition errors were minor or affected function words.

Limitations and Future Work

  • Acoustic Mismatch Remains: While linguistic mismatch was addressed, the paper notes that acoustic differences (accents, pronunciation) are likely larger and harder to overcome, especially with diverse non-native speaker populations.
  • Over-reliance on Prompts: If recognition errors are too severe or affect key concepts, the adaptive prompt system can lead to too many confirmations, causing cognitive overload for non-native users.
  • Future Work:
    • Collect more data to improve acoustic models, especially for diverse non-native populations.
  • Integrate the confirmation prompt generation with state-of-the-art confidence annotation methods.
  • Further explore how to make systems more adaptive to general non-native populations.

Summary updated

BibTeX
@inproceedings{raux2004non,
  author = {Antoine Raux and Maxine Eskenazi},
  title = {{Non-Native Users in the Let’s Go!! Spoken Dialogue System: Dealing with Linguistic Mismatch}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2004},
}
Figure