TL;DR
This paper introduces a novel, cost-effective methodology for evaluating the impact of different interaction strategies in task-oriented conversational agents. By presenting users with curated transcript pairs and asking for pairwise comparisons, the authors enable faster and more controlled evaluation of aspects like empathy and formality, which are often overlooked by traditional methods.
What Problem Does This Paper Address?
Evaluating conversational agents (CAs) has traditionally focused on task completion success (efficiency). However, as CAs become more sophisticated, incorporating human-like interaction capabilities such as empathy or varying formality levels, simply measuring task success is insufficient. These nuanced interaction strategies significantly impact the user's experience (QoE), but evaluating them is challenging. Existing methods often require full system implementation, are time-consuming, or struggle to isolate the effect of specific strategies from confounding variables. This paper aims to bridge this gap by providing a method to evaluate these "softer" aspects of CA interaction efficiently and in a controlled manner.
Methods
The core of the proposed methodology is a within-subject design that leverages pairwise comparison of manually curated transcripts. Instead of users directly interacting with a live agent, they are presented with two slightly different versions of the same interaction transcript. These versions are designed to differ only in the specific interaction strategy being tested (e.g., one formal, one informal).
This approach draws inspiration from:
- Human-Computer Interaction (HCI): Distinguishing between Quality of Service (QoS - efficiency) and Quality of Experience (QoE - how the task is achieved).
- Crowdsourcing: Utilizing platforms like CrowdFlower for efficient data collection.
- A/B Testing: Presenting users with controlled variations to compare.
Key aspects of the method include:
- Controlled Stimulus Material: Transcripts are manually curated by linguists to ensure minimal changes, isolating the variable of interest.
- Pairwise Comparison: Users choose which of the two presented transcripts they prefer, directly indicating their preference for a particular strategy.
- Off-line Evaluation: Eliminates the need for a fully implemented system, allowing for early-stage evaluation and reducing the risk of users missing key interaction elements.
Datasets
The paper does not introduce new datasets. Instead, it uses manually curated transcripts as stimulus material for their experiments. These transcripts are generated based on specific interaction strategies being tested.
Key Insights and Ideas
- QoE is Crucial for Advanced CAs: As CAs evolve beyond simple task completion, evaluating the user experience (QoE) becomes as important as task efficiency (QoS).
- Transcript-Based Evaluation is Viable and Efficient: Presenting users with curated transcripts for pairwise comparison is a fast, cheap, and effective way to evaluate interaction strategies, comparable to direct interaction in some cases.
- Controlled Isolation of Strategies: The methodology allows for the isolation of specific interaction strategies (like empathy or formality) by carefully controlling the transcripts, minimizing confounding factors.
- Crowdsourcing for Nuanced Evaluation: Crowdsourcing platforms can be effectively used to gather subjective judgments on complex interaction qualities, not just task success.
- Gender Differences in Perception: The study revealed interesting gender-based differences in how users perceive certain interaction strategies, particularly alignment and facing.
Key Results
The authors conducted pilot experiments evaluating five interaction strategies: empathy, formality, facing, vocabulary extension, and linguistic alignment.
- Preference for Experimental Conditions: In most cases, subjects preferred the "experimental" condition (where the strategy was applied) over the "control" condition, indicating the methodology successfully captured the impact of these strategies.
- Cost-Effectiveness: The experiment cost approximately $51.48 for evaluating all five variables, with a cost of roughly $10 per variable. The total experiment duration was about 12 hours.
- Empathy, Formality, and Vocabulary Most Appreciated: These strategies showed the most significant positive impact on user preference.
- Alignment and Facing Less Appreciated: These strategies had a less pronounced effect on user preference in this study.
- Impact on QoS/QoE Dimensions: Different strategies had varying impacts on specific QoS and QoE dimensions. For example, alignment primarily affected utility and interaction, while empathy influenced satisfaction and interaction.
Limitations and Future Work
Limitations: * The paper acknowledges that for certain complex strategies (e.g., those involving multiple turns or branching dialog paths), the current guidelines for transcript modification (insertion/substitution) might not be sufficient. * The study used a "between-subject" design for some prior work mentioned, but their proposed method is "within-subject." The paper does not explicitly detail limitations of their within-subject design beyond the need for careful transcript curation.
Future Work: * The authors plan to define and test a wider range of interaction strategies. * They suggest the need for new guidelines and parameters to create proper transcripts for more complex strategies. * The methodology could be used to define constraints for data collection in data-driven systems.
Summary updated