Role of language models in spoken fluency evaluation

Om D. Deshmukh, Harish Doddala, Ashish Verma, Karthik VisweswariahInterspeech 2010

TL;DR

This paper introduces a novel approach to automatically evaluate spoken fluency by leveraging language models (LMs) trained on both "fluent" and "disfluent" speech data. By analyzing the relative perplexity of these LMs on a candidate's speech, the authors demonstrate a significant improvement in classifying fluency levels, outperforming traditional prosodic features alone.

What Problem Does This Paper Address?

Evaluating spoken fluency is crucial for language learning and assessment, particularly in contexts like call centers. Traditional methods often rely on analyzing speech signals directly (prosodic features) or using transcripts generated by Automatic Speech Recognition (ASR) systems. However, these methods can be limited. Prosodic features might not capture subtle linguistic nuances, and ASR-generated transcripts are prone to errors, which can negatively impact the accuracy of lexical feature extraction. This paper tackles the challenge of improving automatic spoken fluency evaluation by exploring how language models, typically used for speech recognition, can be repurposed to directly assess fluency.

Methods

The core innovation lies in training two distinct language models: one on "fluent" speech data and another on "disfluent" speech data. The key idea is that a fluent speaker's utterance should be more predictable (lower perplexity) by the "fluent" LM, while a disfluent speaker's utterance should be more predictable by the "disfluent" LM.

The authors extract features based on the relative perplexity of these two LMs on a given test utterance. Specifically, they consider:

  • Number of fluent-unlikely words (NFU): Proportion of words with low prediction probability from the fluent LM.
  • Number of strong-fluent-unlikely words (NSFU): Proportion of words with low probability from the fluent LM but high probability from the disfluent LM, potentially indicating disfluencies.
  • Reliable relative perplexity (RRP): The difference in perplexities between the two LMs.
  • Relative characteristics words (RCW): A feature that focuses on words with significantly different occurrence counts in the fluent vs. disfluent training data.
  • Global relative perplexity (GRP): The overall difference in perplexities between the two LMs for the entire utterance.

These perplexity-based features are then combined with traditional prosodic features (related to filled pauses and silence) for a comprehensive evaluation.

Datasets

The primary dataset used is the Switchboard database. This database, known for its spontaneous conversations, was annotated for various disfluencies. The authors created "fluent" and "disfluent" versions of transcripts from this database. To address a mismatch between the Switchboard data's content (American activities) and the target evaluation domain (Indian speakers discussing Indian topics), they augmented both datasets with text from Indian news portals.

Key Insights and Ideas

  • Language models as fluency detectors: The paper demonstrates that LMs, typically used for predicting word sequences in ASR, can be effectively repurposed to assess the predictability of speech, which directly correlates with fluency.
  • Complementarity of fluent and disfluent models: Training separate LMs for fluent and disfluent speech allows for a more nuanced understanding of utterance characteristics. The relative performance of these two models is more informative than the performance of a single, general-purpose LM.
  • Perplexity as a fluency indicator: Relative perplexity, especially when considering specific types of "unlikely" words, proves to be a powerful feature for distinguishing between fluent and disfluent speech.
  • Prosodic features are useful, but not sufficient: While prosodic features capture important aspects of disfluency (like hesitations), combining them with the proposed LM-based features leads to a significant performance boost, highlighting the complementary nature of linguistic and acoustic cues.

Key Results

The proposed perplexity-based features achieved classification accuracies of 39.8% for 4-class fluency evaluation and 68.4% for 2-class evaluation. When combined with prosodic features, these accuracies improved to 41.1% for 4-class and 73.7% for 2-class classification. This represents a notable improvement over baseline accuracies and prosodic features alone.

Limitations and Future Work

  • Data bias: The authors acknowledge that the Switchboard data, while useful, has content that differs from the typical topics discussed by Indian speakers. While they attempted to mitigate this by adding Indian news data, a more domain-specific fluent/disfluent dataset would be ideal.
  • ASR errors: The perplexity features are computed on ASR output, which is inherently error-prone. The paper notes that the rate of out-of-vocabulary words in the Switchboard data is around 9%, leading to a high Word Error Rate (WER) of about 50%. Future work aims to address this by learning disfluency patterns to generate synthetic disfluent data for new domains and by refining lexical features to be more robust to ASR errors.
  • Subjectivity of human evaluation: The paper highlights the subjectivity in human fluency assessment, with inter-assessor agreement being only 53.5%, underscoring the challenge of establishing a definitive ground truth.

Summary updated

BibTeX
@inproceedings{deshmukh2010role,
  author = {Om D. Deshmukh and Harish Doddala and Ashish Verma and Karthik Visweswariah},
  title = {{Role of language models in spoken fluency evaluation}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure