Abstract: This paper introduces Regularized Information Gain (RIG), a novel reward mechanism for training agents in goal-oriented visual dialogue. By combining reinforcement learning with information-theoretic constraints, the authors enable agents to ask more strategic, human-like questions that efficiently narrow down an undisclosed object in an image, achieving a new state-of-the-art accuracy of 67.19% on the GuessWhat?! dataset.
Introduction: The Challenge of Strategic Dialogue
In goal-oriented visual dialogue, an agent must identify a specific object in an image by asking a series of "Yes/No" questions to an "Oracle." While humans are naturally adept at using "binary search" strategies to prune a search space—for instance, asking about a category before narrowing down spatial location—artificial agents often struggle with redundancy and a lack of strategic depth.
Existing reinforcement learning (RL) models frequently suffer from two primary issues:
- Redundancy: Asking the same or semantically similar questions multiple times.
- Lack of Strategy: Asking questions that provide little "information gain," failing to distinguish between likely candidates effectively.
The challenge lies in designing a reward function that doesn't just reward the final correct guess, but also encourages the agent to be "conversationally informative" throughout the dialogue.
Methods: Regularized Information Gain (RIG)
The authors propose an end-to-end system based on the Rational Speech Act (RSA) framework, which models human inquiry as a trade-off between the expected information gain of a question and the "cost" of asking it.
The core contribution is the Regularized Information Gain (RIG) reward. It is defined as the expected information gain (measured via KL Divergence between the "prior" and "posterior" object distributions) minus a cost function. Key innovations include:
- Skewness Coefficient ($\beta$): The authors weight the information gain by the skewness of the resulting probability distribution. A more "skewed" distribution means the agent has successfully isolated a small group of likely objects, reducing the chance of a wrong guess later.
- Cost Function ($C(q)$): To prevent the agent from repeating questions, the model imposes a penalty (cost) if a question has been asked before in the same dialogue.
The model was evaluated on the GuessWhat?! dataset, which features 155,280 games across 66,537 images, featuring 134,074 unique objects.
Key Results and Insights
The RIG-based model achieved a state-of-the-art accuracy of 67.19% on the "New Object" task, significantly outperforming the previous baseline of 53.4% and concurrent models. Beyond raw accuracy, the model demonstrated superior efficiency and strategic behavior:
- Reduced Redundancy: Qualitatively, the model was much more efficient. The average number of repetitive questions per dialogue dropped from 0.82 (baseline) to 0.27 (RIG as reward).
- Strategic Questioning: Qualitative analysis showed that the RIG agent follows a logical progression (e.g., asking about the category of an object before asking about its spatial location), whereas baseline agents often asked disjointed or repetitive questions.
- Mimicking Decision Trees: By rewarding questions that lead to skewed distributions, the agent effectively learns a strategy similar to a decision tree, where each question aims to split the remaining "candidate objects" into a very likely group and a very unlikely group.
Conclusion and Future Work
While RIG significantly narrows the gap between artificial agents and human performance, a substantial difference remains (67.19% vs. 84.4%). Future work will focus on extending this framework beyond the "Yes/No/NA" format of the GuessWhat?! game to more open-ended visual dialogues and ensuring that generated questions remain perfectly coherent and diverse in more complex generative settings.
Summary updated