Introduction
Training dialogue agents using reinforcement learning (RL) is often slow and unstable. A primary reason is that user simulators typically provide goals in a completely random order, ignoring the natural progression of learning. This paper introduces ACL-DQN, a framework that employs a "teacher" model to automatically arrange a curriculum of user goals based on the agent's current ability, while incorporating an "over-repetition penalty" to ensure task diversity.
The Challenge of Random Goal Sampling
While RL is the standard for teaching dialogue agents, training against real humans is prohibitively expensive. Researchers use user simulators as a proxy, but these simulators sample "user goals" (e.g., "book a movie ticket for 2 people at 7 PM") randomly. This creates two major hurdles:
- Efficiency: Beginner agents given complex goals fail and learn nothing, while advanced agents gain little from repeating trivial tasks.
- Stability: RL is sensitive to experience order; random sampling leads to high performance variance, hindering real-world deployment.
The ACL-DQN Framework
The authors propose the Automatic Curriculum Learning-based Deep Q-Network (ACL-DQN). Instead of a random simulator, they introduce a Teacher Policy Model that acts as a coach for the Student Agent (the dialogue policy).
Key Components
- The Teacher as an RL Agent: The teacher is an MDP-based agent whose "actions" involve selecting user goals for the student. Its "reward" is tied to the student's learning progress (the increase in success rate).
- Over-repetition Penalty: To prevent the teacher from repeatedly providing easy goals it has already mastered, an Over-repetition Discriminator penalizes the teacher for picking the same goals too frequently, ensuring diversity.
Curriculum Schedules
The researchers evaluated three distinct ways to structure the learning process:
- Schedule A: A single teacher model without predefined difficulty tiers.
- Schedule B: A fixed progression from simple to difficult goals based on time (epochs).
- Schedule C (Mastery-based): The teacher advances the student to harder goals only after a "mastery threshold" (e.g., 50% success rate) is achieved over a specific window.
Experimental Evaluation
The framework was tested on a movie-ticket booking task using 280 annotated dialogues from Amazon Mechanical Turk.
Key Results
- Success Rate: ACL-DQN (Schedule C) reached a success rate of 0.8055 by epoch 400, significantly outperforming the standard DQN baseline of 0.6598.
- Stability: Performance distributions for ACL-DQN variants were much tighter than the baseline, indicating a more stable and reproducible learning process.
- Human Evaluation: In real-user tests, the mastery-based ACL-DQN achieved a success score of 0.526, more than double the baseline DQN's 0.258.
Insights and Future Work
A critical finding was the necessity of the over-repetition penalty; without it, the teacher becomes "lazy," feeding the student easy tasks to maximize its own rewards. Furthermore, mastery-based progression (Schedule C) proved superior to time-based progression (Schedule B), suggesting that curricula must be reactive to the student's actual competence.
Future work will explore more nuanced difficulty metrics beyond "slot count" and test the framework's generalizability across diverse dialogue domains like technical support.
Summary updated