Using quality ratings to predict modality choice in multimodal systems

Ina Wechsung, Klaus-Peter Engelbrecht, Sebastian MöllerInterspeech 2012

TL;DR

This paper explores whether we can predict which input method (speech or touch) users will choose in a multimodal system, and if this prediction can help us estimate the overall system quality. The authors found that quality ratings of individual modalities, particularly "hedonic" (pleasure-related) qualities, are good predictors of modality choice. Crucially, this prediction improves overall system quality estimation, especially when excluding older users.

What Problem Does This Paper Address?

Evaluating the perceived quality of multimodal systems (systems that use multiple input/output methods like speech, touch, gestures) is challenging. While we know that overall system quality is influenced by the quality of its individual components and how often they are used, there's no standardized way to predict which component users will favor. This makes it hard to estimate a system's quality early in development without extensive, complex user studies. This paper aims to bridge this gap by investigating if we can predict modality choice based on the perceived quality of the individual modalities, and if this prediction can then be used to accurately estimate the overall multimodal system quality.

Methods

The core of this research lies in using stepwise linear regression to model modality choice. They collected quality ratings (using the AttrakDiff questionnaire, which measures hedonic and pragmatic qualities) and interaction data (like task duration and abort rates) from users interacting with three different multimodal applications on smartphones.

  • Predicting Modality Choice: They used the quality ratings and interaction parameters from unimodal (single modality) use as predictors to forecast modality choice in the multimodal setting.
  • Predicting Overall System Quality: Once they had a prediction for modality choice (i.e., the proportion of speech vs. touch usage), they used this prediction as a coefficient in a weighted sum of the individual modality quality ratings to predict the overall multimodal system quality. This is based on the idea that the overall quality is a weighted sum of individual modality qualities, with the weight corresponding to usage.

A key methodological aspect is the comparison of results with and without older participants, as prior research suggested age might impact prediction accuracy.

Datasets

The study utilized data collected from three distinct multimodal applications installed on smartphones:

  1. Speech Box Application: A multimodal mailbox system for handling voice messages, emails, and faxes.
  2. Mobile Jukebox Application: An Android-based application for managing music playlists.
  3. IPTV Remote Control Application: An application for controlling an IPTV service on an iPhone.

For each application, data was collected from German-speaking participants. No new benchmark datasets were introduced; instead, data was generated through controlled experiments with these existing applications.

Key Insights and Ideas

  • Hedonic Qualities Dominate Modality Choice: Surprisingly, the "hedonic" qualities (how enjoyable or stimulating a modality is) of individual components were stronger predictors of modality choice than "pragmatic" qualities (efficiency, effectiveness). This suggests users are driven by pleasure and engagement, not just task completion efficiency, when choosing how to interact.
  • Age Matters for Prediction Accuracy: Predictions of modality choice and overall system quality were significantly less precise when older participants (above 55) were included. This highlights a potential challenge in designing universally predictable multimodal systems.
  • Interaction Parameters are Less Predictive: Factors like task duration or abort rates were less effective predictors of modality choice compared to quality ratings. This implies that users' subjective perceptions of quality are more influential than objective performance metrics in their choice of interaction method.
  • Predicting Quality is Possible: The study successfully demonstrated that by predicting modality choice, they could also predict the overall perceived quality of the multimodal system with improved accuracy.

Key Results

  • Modality Choice Prediction: Stepwise linear regression models successfully predicted modality choice for the Jukebox and Remote Control systems, with hedonic qualities (HQI) and perceived effort (SEA) being significant predictors. For the Speech Box, prediction was only successful after excluding older users, with the global attractiveness (ATT) scale becoming a predictor.
  • Overall Quality Prediction: Using predicted modality choice proportions to estimate overall multimodal quality (QMm) resulted in high correlations with actual ratings across all AttrakDiff scales (Hedonic Qualities - Stimulation (HQS), Hedonic Qualities - Identity (HQI), Pragmatic Qualities (PQ), and Attractiveness (ATT)).
  • Improved Prediction with Modality Choice: The prediction accuracy for overall system quality was higher when using predicted modality usage proportions (Equation 1) compared to simply averaging the quality ratings of individual modalities (Equation 2). This difference was statistically significant for HQI and PQ scales.
  • Impact of Excluding Older Users: Excluding older participants consistently led to higher prediction accuracy for modality choice across all systems and overall analyses. For instance, the adjusted R² for the overall system analysis increased from .141 to .395 when older users were excluded.

Limitations and Future Work

  • Lab vs. Real-World Context: The study was conducted in laboratory settings. The authors acknowledge that in real-world scenarios, pragmatic qualities might become more important, and the findings may need verification in more natural usage contexts.
  • Sequential Input Focus: The investigated systems largely offered sequential input. Future work should explore whether these prediction models hold for systems with extensive parallel input capabilities.
  • Indirect Influence of Interaction Parameters: While interaction parameters didn't directly predict modality choice, the authors hypothesize they might have an indirect influence moderated by perceived quality ratings. This warrants further investigation using structural equation modeling.
  • Age-Specific Design: The significant impact of age on prediction accuracy suggests a need for age-aware multimodal system design and evaluation.

Summary updated

BibTeX
@inproceedings{wechsung2012using,
  author = {Ina Wechsung and Klaus-Peter Engelbrecht and Sebastian Möller},
  title = {{Using quality ratings to predict modality choice in multimodal systems}},
  booktitle = {Interspeech},
  year = {2012},
}
Figure