Simulated Chats for Task-oriented Dialog: Learning to Generate Conversations from Instructions

Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, Sachindra Joshi

Abstract

Popular dialog datasets such as MultiWOZ are created by providing crowd workers an instruction, expressed in natural language, that describes the task to be accomplished. Crowd workers play the role of a user and an agent to generate dialogs to accomplish tasks involving booking restaurant tables, calling a taxi etc. In this paper, we present a data creation strategy that uses the pre-trained language model, GPT2, to simulate the interaction between crowd workers by creating a user bot and an agent bot. We train the simulators using a smaller percentage of actual crowd-generated conversations and their corresponding instructions. We demonstrate that by using the simulated data, we achieve significant improvements in low-resource settings on two publicly available datasets - the MultiWOZ dataset and the Persona chat dataset.

Introduction

Building effective task-oriented dialog systems—such as bots that book trains or hotels—requires massive amounts of high-quality conversation logs. Traditionally, these logs are collected via expensive and slow crowdsourcing, where human workers follow specific natural language instructions (e.g., "Book a train to Cambridge arriving by 11:45") to play the roles of both user and agent.

Existing data augmentation methods, such as simple paraphrasing, often fail to capture the complex, multi-turn logic required for task-oriented dialog. This paper introduces a novel data augmentation strategy using two GPT-2 based "simulators"—a User Bot and an Agent Bot—to automate the role-playing process. By generating synthetic conversations, this framework significantly boosts the performance of dialog models in low-resource settings, achieving a 7-13% improvement on the MultiWOZ dataset using only a fraction of the original human-labeled data.

Methodology: Bot-to-Bot Simulation

The core innovation is a simulation framework that mimics the human data collection process through a "Generator + Selector" pipeline.

  1. The User Bot: Driven by task instructions, this bot initiates the conversation and responds to the agent until the goal is met.
  2. The Agent Bot: This bot has access to a Knowledge Base (KB). It parses user requests, queries the database, and provides accurate information like reference numbers or schedules.

To ensure high-quality interactions, each bot employs a two-step process. First, a Response Generator (a fine-tuned GPT-2 model) creates a pool of diverse candidate responses using nucleus sampling. Second, a Response Selector (a Longformer-based model) scores these candidates to select the most relevant one, preventing the repetitive loops common in standard decoding methods.

The bot-to-bot simulation framework showing the interaction between the instruction-driven User Bot and the KB-driven Agent Bot.

Experimental Results

The researchers evaluated the method on two major benchmarks: MultiWOZ (multi-domain task-oriented dialog) and PersonaChat (personality-focused chit-chat). The simulated data provided a substantial performance boost in "low-resource" scenarios where only 5% or 20% of the original human data was available.

On the MultiWOZ dataset, the "Combined Score" (a metric aggregating BLEU, Inform, and Success rates) for the Soloist model jumped from 55.2 to 62.3 in the 5% data setting. Similar gains were observed in the 20% setting, where the score rose from 69.5 to 78.3.

Performance comparison on the MultiWOZ dataset across different data availability percentages (5%, 20%, 100%).

In the PersonaChat dataset, the method improved the Hits@1 metric by 2-10% across different data sizes. A human study further confirmed that while simulated chats were slightly less relevant to instructions than human ones, they were nearly identical in terms of grammar and fluency.

Discussion and Insights

Several key insights emerged from this research:

  • Weak Simulators, Strong Students: Even when simulators are trained on limited data (5-20%), the synthetic conversations they generate are diverse enough to help a "student" model learn better general patterns.
  • Diversity over Greed: The use of nucleus sampling and a Selector module was crucial for moving conversations forward meaningfully rather than producing generic, statistically likely responses.
  • Instruction Following: The study demonstrates that pre-trained models like GPT-2 can be effectively steered via natural language instructions to generate structured, goal-oriented data.

Limitations and Future Work

The method did not improve performance when 100% of the human data was already available, as the model had already learned the necessary patterns and the inherent noise in simulated data provided no additional benefit. Future work may involve using larger language models to improve instruction-following accuracy or exploring noise injection to increase student model robustness.

Summary updated

Abstract

Popular dialog datasets such as MultiWOZ are created by providing crowd workers an instruction, expressed in natural language, that describes the task to be accomplished. Crowd workers play the role of a user and an agent to generate dialogs to accomplish tasks involving booking restaurant tables, calling a taxi etc. In this paper, we present a data creation strategy that uses the pre-trained language model, GPT2, to simulate the interaction between crowd workers by creating a user bot and an agent bot. We train the simulators using a smaller percentage of actual crowd-generated conversations and their corresponding instructions. We demonstrate that by using the simulated data, we achieve significant improvements in low-resource settings on two publicly available datasets - the MultiWOZ dataset and the Persona chat dataset.
BibTeX
@misc{mohapatra2020simulated,
  author = {Biswesh Mohapatra and Gaurav Pandey and Danish Contractor and Sachindra Joshi},
  title = {{Simulated Chats for Task-oriented Dialog: Learning to Generate Conversations from Instructions}},
  year = {2020},
  eprint = {2010.10216},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2010.10216},
}
Figure