Abstract
Goal-oriented visual dialogue requires agents to identify specific objects in an image through sequential questioning. While reinforcement learning (RL) and imitation learning (IL) from human data are common approaches, they often suffer from sparse rewards or limited data coverage. This paper proposes a hybrid training strategy that first imitates analytic experts—Information Gain and Target Posterior experts—and subsequently refines the policy via reinforcement learning. Our results on the GuessWhat?! benchmark demonstrate that this approach allows agents to surpass their own teachers and achieve state-of-the-art performance, particularly in noisy, realistic environments.
1. Introduction
The "Questioner" role in goal-oriented visual dialogue, exemplified by the GuessWhat?! game, involves identifying a secret object in an image by asking a human "Answerer" a series of binary questions. Identifying objects in complex visual scenes requires both visual understanding and strategic dialogue management.
Training these agents is notoriously difficult. Reinforcement learning from scratch is hampered by a massive search space, while imitation learning from human datasets often fails when agents encounter states not covered in training. Information-theoretic approaches like Information Gain (IG) are mathematically sound but "greedy," optimizing only for the immediate next step.
2. Methodology
The researchers proposed a two-phase training pipeline to overcome these limitations.
2.1 Phase 1: Imitation Learning from Analytic Experts
Instead of imitating humans, the agent imitates two probabilistic experts:
- Information Gain Expert (IGE): Maximizes mutual information to effectively partition the candidate object set.
- Target Posterior Expert (TPE): A greedy expert that maximizes the probability of the known target object.
The authors utilized the DAgger algorithm for dataset aggregation, allowing the agent to learn recovery strategies. A Mixture of Experts strategy was employed, using IGE for initial exploration and TPE for final identification.
2.2 Phase 2: Reinforcement Learning Refinement
Because analytic experts rely on an approximate model of the answerer, they are not optimal. The agent's policy is fine-tuned using the REINFORCE algorithm, allowing it to discover long-term strategies that optimize for the final win rate rather than just the next-step information gain.
3. Experimental Results
The model was evaluated on the GuessWhat?! dataset, which consists of 155,000 dialogues and 67,000 unique images.
3.1 Performance and Robustness
In realistic settings (indA), the agent achieved 76.31% accuracy, significantly outperforming the IGE expert it was trained to imitate (71.43%). Furthermore, the model demonstrated superior robustness; while standard experts saw a 10.81% performance drop in noisy environments, the proposed method's performance only decreased by 6.14%.
3.2 Qualitative Analysis
Dialogue observation revealed that the RL-refined model moved away from repetitive questioning and adopted sophisticated spatial reasoning (e.g., "Is it on the bottom row?"), leading to more efficient object discovery.
4. Conclusion
By combining the strengths of analytic experts and reinforcement learning, the authors developed an agent that surpasses its teachers and achieves state-of-the-art performance on the GuessWhat?! benchmark. Future work will explore open-ended question generation and improving efficiency in shorter dialogue sequences.
Summary updated