TL;DR
This paper introduces a novel approach to training dialogue agents by deploying them in a live, role-playing game environment where they interact with intrinsically motivated human players. This "dialogue in the wild" method allows models to continuously learn and improve from real-world interactions, demonstrating more efficient and cost-effective data collection compared to traditional crowdsourcing.
What Problem Does This Paper Address?
Traditional NLP research often relies on static, crowdsourced datasets for training dialogue models. However, this approach has limitations: crowdsourced data may lack naturalness and relevance to real-world use cases, and the static paradigm prevents models from learning from their own conversational experiences. The authors argue for a shift towards systems that can iteratively learn from interactions with humans, mimicking how humans acquire language. The challenge lies in collecting high-quality, diverse, and cost-effective data from real users in a way that facilitates continuous model improvement.
Methods
The core of the paper is the LIGHT (Learning to Interact in a Game with Humans and Bots) system. This system deploys dialogue agents within a fantasy role-playing game. Human players are paired with these agents, and both are assigned characters with personas and backstories. The game is designed with a "mini-game" loop of 6 turns per agent, encouraging engagement.
Key methodological aspects include:
- Lifelong Learning Loop: The system operates in a continuous cycle of collecting human-model conversations, retraining updated models on this new data, and redeploying them.
- Intrinsic Motivation: By framing the interaction as a game with role-playing elements, the authors aim to leverage players' intrinsic motivation rather than payment, leading to more natural and diverse conversations.
- Acting Score Model: An automated "dungeon master" assesses the quality of human role-playing, providing feedback to players and generating a signal for data filtering.
- Retrieval Models: For deployment, the authors primarily use retrieval-based models (Poly-Encoder Transformer) for safety reasons, allowing for pre-vetting of responses. Generative models are also trained and evaluated offline.
- Iterative Data Collection and Training: The system separates dialogue data into (context, next utterance) pairs, using human utterances as training data. They also explore filtering this data based on the acting score.
Datasets
- LIGHT MTurk Data: The initial dataset used for training, consisting of 8.5k episodes and 111k utterances collected via crowdsourcing.
- LIGHT WILD Data: The newly collected dataset from the deployed role-playing game. At the point of freezing, this dataset contained 41,131 episodes and 461,984 utterances. This dataset is characterized by human-model conversations, making it distinct from purely human-human or human-bot datasets.
Key Insights and Ideas
- Intrinsic Motivation Drives Data Quality: Players motivated by the game experience (rather than just payment) generate more natural, diverse, and higher-quality dialogue data.
- "Dialogue in the Wild" is Cost-Effective: Collecting data through the deployed game is significantly cheaper (1/5th the cost per utterance) than traditional crowdsourcing, even after accounting for advertising costs.
- In-Distribution Data is More Effective: Data collected from the deployed system (WILD) is more effective per utterance at improving model performance (e.g., continue rates) than out-of-distribution crowdsourced data.
- Lifelong Learning Yields Continuous Improvement: The iterative training process demonstrably improves model performance across multiple rounds, both on offline metrics and online engagement.
- Model Size vs. Engagement Trade-off: Larger models might make fewer factual errors but can be perceived as more boring, negatively impacting player engagement and thus the quality of future data.
Key Results
- Performance Improvement: Models trained through the lifelong learning setup showed steady increases in performance on metrics like Hits@1/20 across three training rounds. For instance, Hits@1/20 on the WILD test set improved from 81.61% in Round 1 to 87.63% in Round 3.
- Cost Efficiency: The cost per utterance for collecting WILD data was approximately 1/5th of that for LIGHT MTurk data, while yielding better accuracy gains.
- Higher Continue Rates: Models trained on LIGHT+WILD data achieved higher continue rates (75.2%) compared to models trained only on LIGHT MTurk data (74.1%), indicating better human engagement.
- Data Distribution Differences: The WILD dataset exhibits overexpressed words related to aggression, overt friendliness, and adventuring, and increased slang, contrasting with the more "even-keeled" and dry topics found in crowdsourced data.
Limitations and Future Work
- Model Size vs. Boringness: The authors note an "unexpected result" where larger models performed worse in terms of continue rates, potentially due to being perceived as boring. They suggest that future work should focus on understanding and controlling this trade-off.
- Gamification Impact: The paper acknowledges that the specific game design significantly influences player engagement and learning rates, suggesting that further research into game design elements could yield even more cost-effective learning.
- Safety of Generative Models: While retrieval models were used for deployment due to safety concerns, generative models were evaluated offline. Future work could explore safer deployment of generative models.
- Broader Domain Application: The authors suggest their setup can be applied more generally, incorporating dialogue and actions, or situated in other domains beyond the fantasy game world.
Summary updated