Region under Discussion for visual dialog

Mauricio Mazuecos, Franco M. Luque, Jorge Sánchez, Hernán Maina, Thomas Vadora, Luciana BenottiEMNLP 2021 · DOI

Abstract

In Visual Dialog, models must navigate both visual data and conversational history. However, current architectures often treat history as a "black box" of text tokens, leading to poor grounding and a lack of transparency. This paper introduces the Region under Discussion (RuD), a novel spatial representation that explicitly models the "visual common ground" established during a conversation. By defining the RuD as a dynamic bounding box that constrains the search space based on dialog history, we demonstrate significant performance gains on history-dependent questions and provide a more interpretable mechanism for visual reasoning.

Introduction

In the field of Visual Dialog (VD), models are expected to answer questions about an image by considering both the visual content and the preceding conversation. However, it’s often unclear how or even if models are actually using that history. Previous research has shown that many models can perform surprisingly well by ignoring the history entirely, treating the task more like standard Visual Question Answering (VQA).

The authors identify a core gap: the lack of a clear definition of what it means for a question to be "history-dependent." Furthermore, most current models treat dialog history as a sequence of text tokens, a "black box" approach that lacks transparency. This paper aims to define history dependence and create a representation that connects the linguistic history directly to the visual space.

Methods

The authors propose the Region under Discussion (RuD), inspired by the pragmatic theory of "Question under Discussion" (QuD).

  1. Semantic History Construction: They use a semantic parser to extract "intrinsic" properties of the target object (e.g., "it is a car," "it is not a person") from the dialog history.
  2. RuD Generation: Based on these properties, they filter the objects present in the image. The RuD is defined as the smallest bounding box that encloses all candidate objects that haven't been ruled out yet. This box represents the "visual common ground" shared by the participants.
  3. Model Integration: They integrated RuD features into two existing architectures:
  4. QCS (Question+Category+Spatial): A standard baseline where they added RuD-relative spatial coordinates.
  5. CMO (Cross-modal Oracle): An LXMERT-based transformer model where they re-encoded object positions relative to the RuD rather than the whole image.
Architectures for the QCS and CMO models, illustrating how RuD spatial information is integrated into the feature vectors.

Datasets and Evaluation

The research utilizes the GuessWhat?! dataset, a large-scale reference resolution game. To specifically test history dependence, the authors introduced GWHist, a manually annotated subset of 204 questions identified as being truly history-dependent. These are cases where the correct answer changes completely depending on whether the previous dialog is known (e.g., "Is it on the left?" might be 'no' for the whole image, but 'yes' relative to the currently discussed group of cars).

Key Insights and Results

The inclusion of RuD led to massive performance gains on the history-dependent GWHist subset:

  • QCS+RuD improved accuracy from 28.5% to 40.2% (a 41% relative increase).
  • CMO+RuD improved accuracy from 28.5% to 41.6% (a 46% relative increase).

On the full GuessWhat?! test set, the improvements were marginal (around 1%), which confirms the authors' hypothesis that most questions in the dataset don't actually require history. However, for the subset of questions that do require it, RuD is a game-changer.

Qualitative examples showing how the RuD (blue box) constrains the search space, allowing the model to correctly answer history-dependent questions where the baseline CMO model fails.

Interpretability and Grounding

Unlike hidden states in an LSTM or attention weights in a Transformer, a bounding box is immediately interpretable by humans. We can see exactly what part of the image the model thinks is relevant based on the history. Instead of forcing a model to attend to a long string of text, RuD acts as a "visual summary" of the dialog. It tells the model: "Based on what we've said, the target is somewhere in this specific area."

Conclusion

The Region under Discussion provides a powerful framework for grounding dialog history in visual space. By explicitly modeling the spatial constraints imposed by conversation, models can achieve far greater accuracy on complex, context-dependent queries. Future work will explore using Scene Graphs to generate RuDs more dynamically and exploring weakly supervised strategies to learn RuD predictors directly from dialog data without needing manual semantic parsing.

Summary updated

BibTeX
@inproceedings{mazuecos2021region,
  author = {Mauricio Mazuecos and Franco M. Luque and Jorge Sánchez and Hernán Maina and Thomas Vadora and Luciana Benotti},
  title = {{Region under Discussion for visual dialog}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.emnlp-main.390},
}
Figure