SUXES - user experience evaluation method for spoken and multimodal interaction

Markku Turunen, Jaakko Hakulinen, Aleksi Melto, Tomi Heimonen, Tuuli Laivo, Juho HellaInterspeech 2009

TL;DR

Evaluating user experience (UX) in spoken and multimodal systems is tricky. This paper introduces SUXES, a structured method that captures both user expectations before interacting with a system and their actual experiences after, allowing for a direct comparison and deeper understanding of what works and what doesn't.

What Problem Does This Paper Address?

The paper tackles the challenge of reliably evaluating user experience in spoken and multimodal interaction systems. While technical performance metrics are well-established, subjective user evaluations, especially for novel interfaces, are often difficult to standardize and interpret. Existing questionnaires sometimes yield inconsistent results, and there's a growing recognition that "user experience" encompasses more than just basic usability. The authors highlight the need for a method that can systematically capture user expectations and then compare them to their actual experiences, providing a more nuanced view of system performance from the user's perspective.

Methods

The core of the SUXES method is a four-phase, eight-step procedure guided by a web-based wizard. This procedure aims to be semi-automatic and can be conducted remotely or in a lab.

  • Phase 1: Introduction: Gathers background information (demographics, prior experience) and optionally reserves a test slot.
  • Phase 2: Application Introduction: Introduces the system and its modalities without revealing usage instructions, aiming to elicit realistic expectations.
  • Phase 3: Experiment & User Experience: The participant engages with the system to complete tasks. This is followed by an experience questionnaire.
  • Phase 4: Feedback: A final, often custom-designed, questionnaire for additional comments and feedback.

The key innovation lies in the questionnaires used:

  • Background Questionnaire: Collects demographic and domain-specific experience data.
  • Expectations Questionnaire: Participants rate statements about the system/modalities on two scales: an "acceptable level" and a "desired level." This defines a "Zone of Tolerance."
  • Experience Questionnaire: Uses the exact same statements as the expectations questionnaire, but participants rate their perceived level of quality after using the system.

This setup allows for a direct comparison between expected and experienced quality, identifying gaps and areas of satisfaction.

Datasets

Not discussed in the paper. The paper focuses on the methodology and presents case studies using their own evaluations.

Key Insights and Ideas

  • Expectations as Context: Capturing user expectations before interaction is crucial. It provides a baseline for interpreting their experiences and helps understand why certain aspects of the system might be perceived positively or negatively.
  • Zone of Tolerance: The concept of defining an acceptable and desired level of quality creates a "Zone of Tolerance." If perceived experience falls within this zone, it suggests user satisfaction.
  • Subjective vs. Objective Metrics: The paper notes that objective performance metrics (like speed) don't always correlate with subjective user experience. For instance, speech input might be objectively faster but not subjectively preferred if it's not perceived as pleasant or clear.
  • Modality-Specific Insights: SUXES can reveal how different interaction modalities (speech, text, gestures) are perceived relative to user expectations, highlighting strengths and weaknesses for specific applications.
  • Iterative Development Tool: SUXES is presented as a valuable tool for iterative development, providing actionable feedback on what aspects of an application are working well and where improvements are needed.

Key Results

The paper presents results from two case studies: Travelman (route guidance) and Media Center (media playback).

  • Travelman: Domain-specific predictive text input was perceived as superior to speech input, despite speech being objectively faster. This suggests that factors like clarity and ease of use, captured by SUXES, are more critical than raw speed in this context.
  • Media Center: Speech input was highly anticipated and exceeded user expectations, offering a clear advantage over other modalities like keypad and gestures.
  • Other Modalities: Speech output, gestures, and haptic feedback generally ranked low, often failing to meet even the lowest acceptable level, suggesting they didn't offer clear benefits to the tested user groups.
  • Overall Usefulness: Both applications were considered useful, and users expressed willingness to use them in the future, indicating that the core functionality was well-received.

Limitations and Future Work

  • Limitations: While the method is designed for remote use, the presented case studies were conducted in a usability lab with a test conductor. The paper acknowledges that the "neutral" modality (multi-tap text) in the Travelman study might serve as a baseline, but this baseline itself is a result of the specific experiment setup.
  • Future Work: The authors plan to explore experiments with different modalities (e.g., physical browsing), additional dimensions (e.g., joyfulness), diverse user groups, and further automation of the method. They aim to make the process even more automated, potentially removing the need for a human test conductor, which could significantly increase the amount of data collected.

Summary updated

BibTeX
@inproceedings{turunen2009suxes,
  author = {Markku Turunen and Jaakko Hakulinen and Aleksi Melto and Tomi Heimonen and Tuuli Laivo and Juho Hella},
  title = {{SUXES - user experience evaluation method for spoken and multimodal interaction}},
  booktitle = {Interspeech},
  year = {2009},
}
Figure