Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants

Rafael Ferreira, David Semedo, João MagalhãesEMNLP 2024 · DOI

TL;DR

This paper introduces Multi-Trait Adaptive Decoding (mTAD), a novel method for simulating diverse user behaviors in conversational AI systems. By adaptively combining specialized language models for different user traits at decoding time, mTAD offers a flexible and efficient way to create realistic user profiles without extensive fine-tuning, crucial for robust testing and development of conversational assistants.

What Problem Does This Paper Address?

Testing conversational AI systems thoroughly with real users is expensive and time-consuming. User simulators offer a scalable alternative, but accurately capturing the vast diversity of human conversational traits (like cooperativeness, verbosity, or tolerance for errors) remains a significant challenge. Existing methods often struggle to model these traits flexibly or require costly retraining for new user profiles. This paper aims to bridge this gap by proposing a method that can simulate a wide range of user traits and their combinations efficiently and adaptively.

Methods

The core of the proposed solution is Multi-Trait Adaptive Decoding (mTAD). Instead of training a single model for all traits or combining models before inference, mTAD operates at decoding time. It achieves this by:

  1. Trait-Specialized Simulators (STS): For each identified user trait (e.g., engagement, verbosity), a separate language model (LM) is trained. These LMs are fine-tuned using LoRA (Low-Rank Adaptation) to efficiently capture specific trait characteristics without requiring massive computational resources.
  2. Adaptive Decoding: At each step of generating a dialogue, mTAD adaptively combines the outputs (token probabilities) from these specialized LMs. This combination is controlled by tunable weights, allowing for flexible profile creation. The method is "zero-shot" in the sense that it can combine traits without needing new training data for each specific combination.
  3. Dialogue Grounding: The models are trained on real-world conversational data to ensure their behavior is grounded in authentic user interactions. The learning objective is a causal language modeling task, aiming to predict both user intent and utterance.

Datasets

The authors utilized a dataset of 3.6k real-world conversations collected during the Alexa Prize TaskBot Challenge (Gottardi et al., 2022). This dataset focuses on task-oriented dialogues in the cooking and DIY domains, featuring mixed-initiative interactions. They also used a subset of 100 DIY tasks from WikiHow for generalization experiments.

Key Insights and Ideas

  • Specialization is Key: The paper strongly advocates for specialized models (STS) for each user trait. They found that a joint model (JTS) tends to smooth out distributions, making it difficult to capture extreme trait intensities, especially for less common user behaviors.
  • Decoding-Time Flexibility: The mTAD approach's ability to combine trait simulators at decoding time, without retraining, is a major advantage. This allows for rapid creation of diverse user profiles and adaptation to new traits.
  • Trait Interactions Matter: The study highlights that while individual traits can be modeled, their combinations can lead to complex interactions. The mTAD-LA (Level Aware) extension, which distinguishes between dialogue-level and utterance-level traits, shows promise in better capturing these nuances.
  • Real-World Data is Crucial: The reliance on real-world CTA data was essential for identifying relevant traits and grounding the simulator's behavior, revealing differences between naturalistic and crowdsourced data.

Key Results

  • Improved Trait Adherence: Specialized Trait Simulators (STS) generally outperformed Joint Trait Simulators (JTS) in matching reference trait distributions, particularly for extreme trait intensities (Low/High).
  • Generalization: The proposed mTAD framework demonstrated good generalization capabilities to unseen domains (DIY tasks) and models (Vicuna), maintaining its effectiveness.
  • Multi-Trait Performance: mTAD and its extension mTAD-LA showed superior performance compared to a simple sampling baseline in multi-trait combination evaluations. mTAD-LA, which leverages dialogue-level and utterance-level traits, achieved the best results.
  • System Response Quality: The simulated dialogues generally achieved good system response quality (average score over 1.4), with certain traits (high fluency, exploration, emotion) leading to better system performance.
  • Low Degeneration: The generated dialogues exhibited low degeneration rates (≤ 4%), indicating the methods' ability to produce coherent and effective conversations.

Limitations and Future Work

  • Limited Trait Set: The study focused on a specific set of conversational traits. The authors suggest that exploring additional traits could lead to more comprehensive user simulators.
  • Combinatorial Explosion: While mTAD supports flexible combinations, exhaustively exploring all possible trait combinations was beyond the scope of this work.
  • Task Execution Focus: The current simulator primarily focuses on the task execution phase. Future work could extend this to the retrieval and grounding stages of conversational assistants.
  • Broader Understanding: The authors believe further research is needed to achieve a more comprehensive understanding of user simulation dynamics.

Summary updated

BibTeX
@inproceedings{ferreira2024multi,
  author = {Rafael Ferreira and David Semedo and João Magalhães},
  title = {{Multi-trait User Simulation with Adaptive Decoding for Conversational Task Assistants}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2024},
  doi = {10.18653/v1/2024.findings-emnlp.945},
}
Figure