Abstract
Popular dialog datasets such as MultiWOZ are created by providing crowd workers an instruction, expressed in natural language, that describes the task to be accomplished. Crowd workers play the role of a user and an agent to generate dialogs to accomplish tasks involving booking restaurant tables, calling a taxi etc. In this paper, we present a data creation strategy that uses the pre-trained language model, GPT2, to simulate the interaction between crowd workers by creating a user bot and an agent bot. We train the simulators using a smaller percentage of actual crowd-generated conversations and their corresponding instructions. We demonstrate that by using the simulated data, we achieve significant improvements in low-resource settings on two publicly available datasets - MultiWOZ dataset and the Persona chat dataset.
Introduction: The Data Bottleneck
Building a conversational AI that can help you book a train or find a hotel is a massive undertaking. Traditionally, this requires thousands of "gold standard" conversation logs. To get them, researchers usually hire hundreds of crowd workers to role-play: one person acts as the customer, and the other acts as the travel agent. As you can imagine, this is incredibly slow, expensive, and difficult to scale.
In their EMNLP 2021 paper, researchers from IBM Research and IIIT Bangalore propose a clever shortcut: Simulated Chats. Instead of hiring humans to talk to each other, why not train two AI models to role-play the conversation based on a set of instructions?
Methodology: The Simulated Chat Framework
The researchers built a "Teacher-Student" framework. First, they train "Teacher" bots on a small amount of human data (as little as 5%). These teachers then generate a massive synthetic dataset, which is used to train the final "Student" model.
The simulation involves two distinct bots:
- The User Bot: This bot is given a "goal" (e.g., "Find a train to Cambridge arriving by 11:45"). It uses GPT-2 to generate several possible responses and a Longformer-based selector to pick the most relevant one.
- The Agent Bot: This bot has access to a Knowledge Base (KB). It first generates a "belief state" (a database query), retrieves results (e.g., how many trains are available), and then generates a response to the user.
Instead of just taking the first thing the AI says (greedy decoding), the bots generate a "pool" of candidate responses using nucleus sampling. A separate selector model then picks the best response from that pool. This ensures the conversation stays on track while remaining diverse.
Experimental Results
The authors evaluated their approach on two major benchmarks: MultiWOZ (2.0 and 2.1) and PersonaChat. The results were particularly impressive in "low-resource" scenarios where data is scarce.
On the MultiWOZ dataset, using only 5% of the original training data, the addition of simulated data led to a 7-13% improvement in the "Combined Score," which measures task completion and information accuracy.
In a blind study, humans rated the simulated chats nearly as high as real human chats. On a scale of 1-5, humans gave the simulated chats a 4.5 for grammar and a 4.1 for fluency (compared to 4.6 and 4.5 for real humans).
Discussion and Conclusion
The proposed method offers significant cost-efficiency. The authors estimate that generating 10,000+ dialogs using their method costs less than $400** in cloud computing time, whereas hiring crowd workers for the same task would cost roughly **$7,500.
While the results are a major step forward, simulation didn't help much when 100% of the human data was already available. It seems simulation is best used to fill gaps in low-resource settings. Human evaluators also noted that simulated bots occasionally ignore specific details in instructions, scoring slightly lower on relevance (4.0) than humans (4.7). Future work will explore making simulators even more diverse and comparing this method against other advanced augmentation techniques like GANs or Reinforcement Learning.
Paper Citation: Biswesh Mohapatra, Gaurav Pandey, Danish Contractor, and Sachindra Joshi.
- Simulated Chats for Building Dialog Systems: Learning to Generate Conversations from Instructions. In Findings of the Association for Computational Linguistics: EMNLP 2021.
Summary updated