Differences in User Responses to a Wizard-of-Oz versus Automated System

Jesse Thomason, Diane LitmanNAACL 2013

TL;DR

This paper investigates whether users behave differently when interacting with a dialogue system that is secretly controlled by a human (Wizard-of-Oz, WOZ) versus a fully automated system. They found significant differences in user speech patterns, suggesting that WOZ data, commonly used to train automated systems, may not be as reliable as previously assumed.

What Problem Does This Paper Address?

When developing automated dialogue systems, researchers often use the "Wizard-of-Oz" (WOZ) methodology. In this setup, a human secretly controls parts of the system (like speech recognition or response generation) while the user believes they are interacting with a fully automated system. The goal is to collect realistic user interaction data to train and improve the final automated system. However, a key assumption is that user behavior in a WOZ setup is similar to behavior with a truly automated system. This paper questions that assumption, specifically asking if the presence of a hidden human operator, even if undetectable, influences how users speak and interact. This is crucial because if WOZ data systematically differs from automated system data, it could lead to suboptimal training of automated systems.

Methods

The researchers conducted a post-hoc analysis of data from two prior experiments that used a spoken tutorial dialogue system. They compared user responses from a WOZ condition against a fully automated (AUT) condition. The core of their method involved analyzing user turns using two types of features:

  • Lexical Features: They used the Linguistic Inquiry and Word Count (LIWC) tool to analyze the percentage of words falling into various categories (e.g., tentative words, prepositions). They extracted 69 LIWC categories from human transcriptions of user turns.
  • Prosodic Features: They extracted features like duration, pitch, and energy (RMS) from the audio recordings of user speech using the openSMILE toolkit. They calculated maximum, minimum, mean, and standard deviation for pitch and energy, along with speech duration and pause length, resulting in 10 prosodic features.

To account for variations in recording environments and individual speaking styles, prosodic features were normalized by their value in the user's first turn of a dialogue.

They then employed two main analytical approaches:

  1. Statistical Comparison: A Welch's two-tailed t-test was used to compare the distributions of each feature's values between WOZ and AUT responses on a per-question basis.
  2. Machine Learning Classification: They trained J48 decision tree models on each question to classify whether a user response originated from the WOZ or AUT experiment. This was done using 10-fold cross-validation. Models were evaluated against a majority-class baseline.

To control for inherent differences between the student groups in the original experiments, they also performed a random splitting procedure and tested for feature differences between these new groups.

Datasets

The study used data collected from two prior experiments involving a spoken tutorial dialogue system. The system tutored university students in basic Newtonian physics.

  • WOZ Condition: 21 users, 111 unique questions, 1542 total turns.
  • AUT Condition: 25 users, 111 unique questions, 2034 total turns.

The data consisted of audio recordings of student speech and human transcriptions of these utterances. No new datasets were introduced; the analysis was performed on existing experimental data.

Key Insights and Ideas

  • WOZ data isn't a perfect proxy for AUT data: The most significant finding is that user responses do differ between WOZ and AUT systems, even when the system's core functionality (beyond the wizard's role) is identical and users are unaware of the wizard. This challenges the common assumption that WOZ data is directly interchangeable with data from a fully automated system.
  • Subtle differences matter: The differences weren't just in what users said (lexical content) but also how they said it (prosody). This suggests that the perceived automation level, even if subtle, impacts user expression.
  • User confidence might be key: The authors hypothesize that differences in user confidence, potentially influenced by the perceived reliability of the system, drive these behavioral changes. Users might become less confident over time with an automated system, leading to altered speech patterns.
  • Machine learning can detect the difference: The fact that J48 decision trees could classify the experiment-of-origin for a significant portion of the turns (32.79%) based on prosodic and lexical features alone demonstrates that these differences are systematic and detectable.
  • Hyper-articulation in AUT: In some cases, users in the AUT condition exhibited hyper-articulation (speaking more slowly, loudly, and enunciating more clearly). This might be a strategy to compensate for perceived system errors or to ensure better speech recognition, contributing to the classifiers' ability to distinguish between setups.

Key Results

  • Significant feature differences: After controlling for between-student variability, all 10 prosodic features and 29 out of 69 lexical features differed significantly (p < 0.05) for at least one question.
  • Classifiable responses: J48 decision tree models could classify the experiment-of-origin for 21 out of 97 questions (32.79% of the corpus by turns) with statistical significance (p < 0.05).
  • Specific feature examples:
    • Features like "Duration" and "RMS mean" were strong indicators in classifying responses.
  • The LIWC category "pronoun" showed higher percentages in WOZ utterances, while "Dictionary Words" also showed a trend towards WOZ.
  • In one example, users in the AUT setup were more likely to give shorter responses ("curt") to the question "Would you like to do another problem?".
  • In another example, users in the AUT setup tended to speak louder (higher RMS mean) when responding to a physics question.

Limitations and Future Work

  • Wizard's role was limited: The study's WOZ setup was limited to evaluating speech recognition and correctness. The authors acknowledge that a wizard with a broader role might elicit different user behaviors.
  • Potential for user confidence shift: The observed differences might be more pronounced later in the dialogue as users' confidence in the automated system is tested.
  • Normalization methods: The specific normalization method used for prosodic features might influence the visibility of differences. Exploring alternative normalization techniques is suggested.
  • Future Directions:
    • Deeper exploration: Investigate the nature of differences more deeply, particularly how they evolve over the course of a dialogue.
  • Intentional wizard error: Introduce intentional errors in a wizarded system to simulate frustration and study user responses to a less reliable system.
  • Post-hoc domain adaptation: Explore techniques to adapt WOZ data to better match data seen by automated systems, making WOZ data more useful for training.

Summary updated

BibTeX
@inproceedings{thomason2013differences,
  author = {Jesse Thomason and Diane Litman},
  title = {{Differences in User Responses to a Wizard-of-Oz versus Automated System}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure