Say it as you mean it - analyzing free user comments in the VOICE awards corpus

Florian Gödde, Sebastian MöllerInterspeech 2010

TL;DR

This paper analyzes free-text user comments from the German VOICE Awards corpus to understand what aspects of spoken dialog systems (SDSs) users find most important and how these relate to overall system ratings. They found that while ASR and TTS issues are frequently mentioned, factors like efficiency and dialog structure have a stronger impact on user satisfaction, suggesting a user tradeoff between obvious interface issues and core usability.

What Problem Does This Paper Address?

Traditional usability questionnaires for spoken dialog systems (SDSs) often rely on quantitative scales (e.g., Likert scales) to measure user satisfaction. While these provide a numerical score, they fail to capture the "why" behind a user's opinion. Free-text comments, on the other hand, offer rich qualitative insights into user experiences but are labor-intensive to analyze. This paper addresses the challenge of systematically analyzing these free-text comments to uncover which aspects of SDSs are most critical to user satisfaction and how these qualitative insights correlate with quantitative ratings and actual system performance metrics.

Methods

The researchers analyzed free-text comments from the "What was remarkably positive about the system?" and "What was remarkably negative about the system?" sections of questionnaires collected during the German VOICE Awards competition from 2006-2008. They developed a categorization scheme to group similar comments into predefined aspects (e.g., ASR, TTS, Efficiency, Dialog Structure, Kindness). These categorized statements were then statistically analyzed to determine their frequency and their correlation with overall system ratings and dialog parameters like dialog length and error count. Finally, they used these findings to build a general linear model (GLM) to predict overall user ratings based on categorized user feedback.

Datasets

The study utilized the German VOICE Awards corpus, specifically data collected from 2006 to 2008. This corpus consists of dialogs and questionnaires from user tests of commercial German spoken dialog systems across various domains (e.g., telephone banking, travel information, customer support). The analysis focused on ratings from laypersons. The paper does not introduce a new benchmark dataset but rather analyzes an existing one.

Key Insights and Ideas

  • User Tradeoffs: Users seem to make a conscious tradeoff between obvious interface issues (like ASR or TTS quality) and core usability factors (like efficiency and dialog structure). While ASR and TTS are frequently mentioned as negative aspects, they don't always have the strongest influence on overall ratings.
  • Efficiency and Dialog Structure Reign Supreme: Aspects like system efficiency and a clear dialog structure appear to have a more significant positive impact on user satisfaction than perceived ASR or TTS quality. Users who found systems easy to navigate and efficient to use tended to rate them higher.
  • Kindness Matters: The "kindness" of the system (how it interacts with the user) also emerged as a significant factor influencing overall ratings.
  • Predictive Power of Qualitative Data: Free-text comments, when systematically categorized, can provide valuable insights for predicting user satisfaction, even outperforming models that rely solely on quantitative metrics.

Key Results

  • Most Mentioned Positive Aspects: Good speech recognition (ASR) was the most frequently mentioned positive aspect in 2006, while dialog efficiency took the lead in 2007 and 2008. Other important positive factors included dialog structure, TTS quality, kindness, and learnability.
  • Most Mentioned Negative Aspects: Poor ASR and TTS quality were the most common negative comments, followed by unclear dialog structure, bad efficiency, and task failure.
  • Influence on Ratings: Positive ASR, kindness, and efficiency were the strongest positive influencers on overall ratings. Conversely, negative efficiency, dialog structure, and task completion issues had a significant negative influence.
  • Dialog Length and Efficiency: Users who criticized efficiency had significantly longer dialogs, while those who complimented it had shorter ones.
  • ASR Errors: In 2008, users who praised ASR performance had fewer errors per dialog (0.6) compared to those who criticized it (1.9). This correlation was not statistically significant in 2007.
  • GLM Performance: A general linear model built on categorized feedback and dialog parameters showed promise in predicting overall ratings, though it struggled with extreme ratings (e.g., 1 or 6).

Limitations and Future Work

  • Limitations: The authors acknowledge that the GLM produced slightly too negative ratings for highly-rated systems and too positive for poorly-rated ones, suggesting extreme ratings are still hard to predict. They also note that the statistical significance of some findings (like ASR errors in 2007) was limited by sample size. The analysis of TTS quality was not deeply explored due to its complexity.
  • Future Work: The authors suggest weighting statements based on their perceived impact (e.g., "never understood me" vs. "did not understand me a few times") to improve predictive models. They also plan to use these findings to improve models for predicting user satisfaction, potentially by weighting dialog parameters based on their perceived importance. Further research could explore how to better account for the variance in ratings using these qualitative insights.

Summary updated

BibTeX
@inproceedings{godde2010say,
  author = {Florian Gödde and Sebastian Möller},
  title = {{Say it as you mean it - analyzing free user comments in the VOICE awards corpus}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure