Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History

Tianai Dong, Alberto Testoni, Luciana Benotti, Raffaella BernardiACL 2021

TL;DR

This paper introduces VISPA, a new dataset of visually grounded spatial questions that crucially depend on dialogue history for correct interpretation. They propose a "trigger-zoomer" methodology to automatically identify these context-dependent questions and evaluate existing multimodal models, finding that current architectures struggle to effectively leverage dialogue history.

What Problem Does This Paper Address?

Visual Question Answering (VQA) has evolved into Visual Dialogue, where agents must not only understand an image and a question but also maintain conversational context. While VQA focuses on grounding a single question to an image, Visual Dialogue requires grounding against both the image and the dialogue history. Existing Visual Dialogue datasets, like VisDial, have a limited number of questions that truly necessitate dialogue history for interpretation. This paper addresses the gap of creating and evaluating models on a dataset specifically designed to capture these history-dependent spatial questions, which are crucial for more natural and intelligent conversational agents.

Methods

The authors propose a "trigger-zoomer" methodology to identify history-dependent follow-up questions (FuQs). A "trigger" question establishes a context (e.g., identifying a group of objects), and a "zoomer" question then refines this context by asking about a specific subset or member of that group. They automatically extract potential trigger-zoomer pairs based on linguistic patterns (e.g., group questions followed by other group questions, or color questions followed by group questions). These automatically extracted pairs are then manually annotated to filter for those that genuinely require dialogue history for correct interpretation. They also evaluate several baseline and state-of-the-art multimodal models (LSTM, V-LSTM, LXMERT) on their newly created dataset, both with and without access to dialogue history.

Datasets

The primary dataset used for evaluation is GuessWhat?! (de Vries et al., 2017), a multimodal dataset where dialogues aim to identify a referent in an image. The authors extract a subset of questions from the test set of GuessWhat?!. They also introduce the VISPA dataset, which is a manually filtered subset of context-dependent questions derived from GuessWhat?!.

Key Insights and Ideas

  • The "Trigger-Zoomer" Phenomenon: Many spatial questions in visual dialogues aren't standalone; they build upon previous turns, progressively narrowing down the focus. This "trigger-zoomer" pattern is a key indicator of history dependence.
  • Spatial Questions are Tricky: Spatial questions, especially those involving relative positioning or group membership, are more likely to be history-dependent than other types of questions.
  • Current Models Don't Fully Leverage History: Despite advancements in multimodal architectures, state-of-the-art models like LXMERT still struggle to effectively utilize dialogue history for answering context-dependent questions. Their performance doesn't significantly improve when history is provided, and in some cases, they even show increased uncertainty when they make mistakes.
  • Attention Doesn't Tell the Whole Story: Analyzing LXMERT's attention mechanisms reveals that the model doesn't necessarily focus more on the "trigger" question when answering a context-dependent "zoomer" question, suggesting a failure to properly integrate the historical context.

Key Results

  • The authors successfully created the VISPA dataset, containing 271 manually verified context-dependent questions.
  • In experiments on the GuessWhat?! dataset, LXMERT achieved the highest accuracy overall. However, when evaluated on context-dependent questions, its accuracy dropped significantly.
  • Providing dialogue history (DH) to models did not consistently lead to statistically significant improvements in accuracy, except for the LSTM model on absolute questions. This suggests models are not reliably learning from the history.
  • LXMERT-DH showed higher confidence in correct answers but also exhibited more uncertainty when it failed, indicating a partial awareness of its limitations.

Limitations and Future Work

  • Limited Scope of Methodology: The "trigger-zoomer" methodology, while effective, might not capture all forms of history dependence, especially in longer, open-world dialogues.
  • Generalizability: The applicability of the trigger-zoomer method to dialogues beyond referential games and to other modalities (e.g., captions as triggers) needs further investigation.
  • Model Architecture: The authors suggest that current Oracle architectures are not designed to effectively exploit dialogue history. Future work should focus on designing models that are explicitly trained to attend to and integrate dynamically changing multimodal contexts.
  • Confidence vs. Accuracy: While LXMERT-DH showed increased confidence in correct answers, the lack of consistent accuracy improvement with history suggests a need for deeper investigation into how models learn and express certainty in context-dependent scenarios. The paper poses the problem of interpreting follow-up questions as an open challenge for the community.

Summary updated

BibTeX
@inproceedings{dong2021visually,
  author = {Tianai Dong and Alberto Testoni and Luciana Benotti and Raffaella Bernardi},
  title = {{Visually Grounded Follow-up Questions: a Dataset of Spatial Questions Which Require Dialogue History}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure