On the role of effective and referring questions in GuessWhat?!

Mauricio Mazuecos, Alberto Testoni, Raffaella Bernardi, Luciana BenottiACL 2020

TL;DR

This paper introduces two novel metrics, "effectiveness" and "referring," to better evaluate the quality of questions asked in referential visual dialogue games like GuessWhat?!. They find that while models can achieve high task success, they don't necessarily ask more "effective" questions, and unlike humans, they rarely use "referring" questions to confirm their guesses before making them.

What Problem Does This Paper Address?

Referential visual dialogue systems, where an agent (Questioner) must identify a target object in an image by asking questions to another agent (Oracle), are typically evaluated solely on task success – whether the correct object is identified. However, this metric doesn't capture the quality of the dialogue itself or how efficiently the questions contribute to the goal. The authors argue that we need finer-grained evaluation tools to understand the underlying communication strategies and identify areas for improvement, especially as these systems become more sophisticated and are deployed in real-world scenarios.

Methods

The paper proposes two new metrics for evaluating questions:

  • Effectiveness: A question is considered "effective" if it rules out at least one incorrect object (distractor) from the set of possible referents. This is measured by tracking the set of candidate objects that remain plausible after each question and answer.
  • Referring: A question is "referring" if it uniquely identifies the target object among all objects in the image. This means the Oracle would answer "yes" only for the referent object and "no" for all others.

These metrics are then applied to analyze dialogues generated by state-of-the-art models on the GuessWhat?! dataset, comparing them to human dialogues.

Datasets

The primary dataset used is GuessWhat?! (de Vries et al., 2017). This dataset consists of human-human dialogues generated from images, where one player is given a target object and the other asks yes/no questions to identify it. The dataset contains over 150,000 dialogues with an average of 5.3 questions.

Key Insights and Ideas

  • Beyond Task Success: The paper strongly advocates for moving beyond simple task success metrics to evaluate the quality and efficiency of communication in visual dialogue.
  • Effectiveness is Not Enough for Success: Surprisingly, successful dialogues (both human and model-generated) did not necessarily have a higher percentage of "effective" questions. This suggests that other strategies, not solely focused on eliminating distractors, contribute to success.
  • Humans Confirm Before Guessing: Humans tend to use "referring" questions at the end of a dialogue to confirm their guess before making it. This strategy is linked to higher task success.
  • Models Lag in Confirmation: State-of-the-art models, in contrast to humans, rarely employ these confirming "referring" questions, even in successful dialogues.
  • Dialogue History Matters: The authors highlight the importance of models considering dialogue history, noting that some previous systems did not adequately account for it.

Key Results

  • Effectiveness Decreases Over Time: For both humans and models, the percentage of effective questions tends to decrease as the dialogue progresses, reaching its lowest point in the final turn.
  • Human Effectiveness Higher: Human dialogues generally exhibit a higher percentage of effective questions compared to most models, though the VDST model comes close.
  • Referring Questions and Task Success: Human dialogues that end with a referring question achieve a task success rate of 95%, significantly higher than the 80% success rate for dialogues that do not.
  • Model Referring Question Rate Low: The best-performing model in the study only used referring questions in about 7% of its dialogues, while humans used them in 26% of theirs.

Limitations and Future Work

  • Oracle Accuracy: The authors acknowledge that their metrics rely on an automatic "Oracle" to determine answers. While this Oracle is accurate on human-generated questions, they note that machine-generated questions might be easier for it, potentially skewing results. Future work will investigate this further.
  • Refining "Referring" Metric: The definition of a "referring" question might need refinement. The paper notes that some human confirmation questions, while seemingly referring, might not be strictly so by their definition (e.g., "the one near us on the right?" could also apply to a non-referent object).
  • Model Learning: A key future direction is to explore whether models can be trained to adopt human-like confirmation strategies, specifically using referring questions before guessing, to improve task success.
  • Heuristics for Training: The authors believe their proposed metrics can serve as valuable heuristics to guide the training of end-to-end dialogue models.

Summary updated

BibTeX
@inproceedings{mazuecos2020role,
  author = {Mauricio Mazuecos and Alberto Testoni and Raffaella Bernardi and Luciana Benotti},
  title = {{On the role of effective and referring questions in GuessWhat?!}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2020},
}
Figure