The influence of expertise and efficiency on modality selection strategies and perceived mental effort

Ina Wechsung, Stefan Schaffer, Robert Schleicher, Anja Naumann, Sebastian MöllerInterspeech 2010

TL;DR

This paper investigates how the efficiency of input methods (speech vs. virtual keyboard) influences user choices and perceived mental effort in multimodal systems. They found that users switch to speech when it offers a shortcut of at least 3-4 interaction steps, regardless of their expertise level.

What Problem Does This Paper Address?

Multimodal systems, which allow users to interact using multiple input methods (like speech and touch), are generally assumed to improve user experience. However, it's not always clear why users choose one modality over another, or if they even utilize the multimodal capabilities effectively. Previous research suggests users often stick to a single modality, even when multimodality could be beneficial. This paper aims to pinpoint the specific efficiency threshold that drives users to switch modalities, and how this choice impacts their mental effort, considering the role of user expertise.

Methods

The researchers designed a study using a mobile jukebox application on an Android smartphone. Participants could interact using either a virtual keyboard or speech input. The core of the experiment involved a series of search tasks where participants had to find artists. Crucially, the length of the artist names increased incrementally with each task, creating a predictable increase in the number of interaction steps required for the virtual keyboard. This allowed the researchers to systematically vary the "shortcut" potential of speech input.

They measured:

  • Modality Selection Strategy: Which input method was chosen for each task.
  • Number of Attempts: How many times a participant had to try before successfully completing a task.
  • Perceived Mental Effort: Assessed using the Subjective Mental Effort (SEA) scale.
  • Expertise: Participants were categorized as either experts (experienced with speech dialogue systems) or novices.

Datasets

Not discussed in the paper. The study used a custom-built mobile application for data collection.

Key Insights and Ideas

  • Efficiency Trumps Expertise (Mostly): While expertise plays a minor role in specific tasks, the primary driver for switching to speech input is the efficiency gain offered by speech as a shortcut.
  • The "Magic Number" for Shortcuts: The study identified a critical threshold: when speech input can save users at least 3-4 interaction steps compared to the virtual keyboard, they are significantly more likely to choose speech.
  • Consistency is Key: Once a user switches to a modality (especially speech), they tend to stick with it for subsequent tasks, suggesting a preference for internal interaction consistency.
  • Mental Effort Tied to Success: Perceived mental effort is more strongly correlated with task success (or failure) than with user expertise. If a task is difficult or requires many attempts, users report higher mental effort.
  • Novices are Quick Learners: Novice users demonstrated a clear modality switch after a certain number of interaction steps, and interestingly, none of them required a fourth attempt to switch modalities, suggesting they adapted quickly.

Key Results

  • Modality Switch Threshold: For general users, the threshold for switching to speech was found to be around 3-4 skipped interaction steps. For experts, this threshold was slightly lower, around 3 steps.
  • Speech Preference: Speech usage significantly increased from Task 4 (5 characters) onwards, aligning with the identified shortcut threshold.
  • Mental Effort: While overall mental effort didn't differ significantly between experts and novices, specific tasks (Task 6, 7 characters) were perceived as more demanding. Novices reported higher mental effort than experts on tasks 7 and 10.
  • Number of Attempts: While overall attempts were similar, experts surprisingly needed more attempts on tasks 3 and 9. Task 6 required the most attempts overall.
  • Modality Switching: Participants were hesitant to switch modalities after just one unsuccessful attempt. A majority switched after the second unsuccessful attempt, and nearly all switched after the third. Novices never needed a fourth attempt.

Limitations and Future Work

  • Predictable Task Difficulty: The constant, incremental increase in artist name length made the task difficulty predictable. Future work should randomize task difficulty to see if the results hold.
  • Virtual Keyboard Only: The study only compared speech to a virtual keyboard. Future research should include physical keyboards, as their usability might alter the modality selection threshold.
  • Limited Error Variation: The study had zero tolerance for virtual keyboard errors and relatively good speech recognition accuracy (~77%). Future work should systematically vary error rates to understand the interplay between efficiency, robustness, and modality choice.
  • Input Type: The study focused on text input (artist names). Different input types, like numbers, might yield different results, as numbers are less likely to be entered via speech according to prior research.

Summary updated

BibTeX
@inproceedings{wechsung2010influence,
  author = {Ina Wechsung and Stefan Schaffer and Robert Schleicher and Anja Naumann and Sebastian Möller},
  title = {{The influence of expertise and efficiency on modality selection strategies and perceived mental effort}},
  booktitle = {Interspeech},
  year = {2010},
}
Figure