TL;DR
This paper addresses the lack of standardized parameters for evaluating multimodal human-computer interaction (HCI) systems. The authors propose an extension to existing spoken dialogue system parameters and demonstrate their utility through three usage scenarios, including modeling user satisfaction and identifying system issues.
What Problem Does This Paper Address?
Evaluating multimodal interaction systems is challenging because there's no universally agreed-upon set of metrics to quantify their performance. While spoken dialogue systems have established parameters (like dialog duration or word error rate), the addition of new modalities (like gestures or touch input) complicates this. Researchers often create their own ad-hoc evaluation schemes, making it difficult to compare results across different studies or to systematically improve system modules. This paper aims to bridge this gap by proposing a comprehensive set of multimodal interaction parameters and showing how they can be used for systematic evaluation.
Methods
The authors extend existing interaction parameters from spoken dialogue systems to accommodate multimodal inputs and outputs. For example, "words per turn" becomes "elements per turn," where an element can be a word, gesture, or GUI interaction. They also introduce new parameters specifically for multimodality, such as "fusion accuracy" (how well different modalities are combined) and "modality changes" (how often users switch between modalities).
To demonstrate the practical application of these parameters, they conducted an experiment with a multimodal smart-home system. This system allowed control via voice, gestures (using a smartphone's accelerometer), and touch input on a smartphone GUI. They collected detailed log data and used video annotation to extract the proposed parameters. The experiment focused on three usage scenarios: modeling user ratings (using a regression model), identifying system problems, and analyzing user interaction patterns.
Datasets
The paper does not explicitly mention using pre-existing datasets for training. Instead, it describes a newly conducted experiment with 24 participants interacting with a multimodal smart-home system. The data collected from this experiment (textual and video logs, along with questionnaire responses) serves as the benchmark for evaluating the proposed interaction parameters.
Key Insights and Ideas
- Need for Standardization: The core idea is that a standardized set of parameters is crucial for systematic evaluation and comparison of multimodal systems, similar to what exists for unimodal systems.
- Extending Existing Frameworks: The paper shows that many parameters from spoken dialogue systems can be adapted, but new ones are essential to capture the nuances of multimodal interaction.
- Multimodality-Specific Metrics: Parameters like "fusion accuracy" and "modality changes" are vital for understanding how well different input channels work together and how users navigate them.
- Predictive Evaluation: The proposed parameters can be used within frameworks like PARADISE to predict user satisfaction and identify areas for improvement.
- User Preferences: The study highlights that users often prefer modalities they are familiar with (like touch) and that system design (like response delay) significantly influences modality choice.
Key Results
- Multimodal Accuracy: The smart-home system achieved high recognition rates for spoken input (100% due to a wizard replacement) and touch input, but gesture recognition was lower (42% on average). Combined smartphone input (touch and gesture) achieved 85.62% accuracy.
- User Turns: There was high variability in the number of turns per modality, with touch input being used most frequently (average of 30.92 turns).
- System Response Delay (SRD): The average SRD was 1.59 seconds. A positive correlation was found between SRD and the number of spoken turns, indicating that the speech recognizer replacement negatively impacted spoken interaction speed.
- Modality Changes: Participants changed modalities an average of 11.87 times, showing a dynamic interaction pattern.
- User Ratings: A regression model using interaction parameters explained about 50% of the variance in user ratings, with "number of user turns via spoken input" having a positive impact, and "incorrectly parsed user utterances" and "dialog duration" having negative impacts.
- Problem Identification: The study successfully identified issues like low gesture recognition accuracy and high system response delay for spoken input, which were then addressed.
Limitations and Future Work
- Parameter Applicability: The authors acknowledge that not all proposed parameters are relevant for every multimodal system.
- User Characteristics: The study notes that user characteristics and hedonic aspects of the system also influence perceived quality, which were not fully captured by the interaction parameters alone.
- Annotation Schemes: Developing suitable and efficient annotation schemes for extracting multimodal interaction parameters remains an ongoing challenge.
- Future Work: The authors suggest further investigation into correlations between multimodal parameters and user strategies/characteristics. They also emphasize the need for robust annotation schemes that incorporate task information (start, end, success) for a more complete analysis.
Summary updated