A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions

Takuma Udagawa, Takato Yamazaki, Akiko AizawaEMNLP 2020 · DOI

Abstract

While visual dialogue models often perform well on standard benchmarks, it remains unclear whether they truly understand linguistic grounding or merely exploit dataset biases. This paper presents a fine-grained linguistic analysis of spatial expressions within the OneCommon Corpus. By introducing a structured annotation schema and a novel model, NUMREF, the authors identify a significant "What vs. Where" gap in current AI capabilities. Results show that while models handle object attributes reasonably well, they struggle with inter-entity spatial relations and linguistic ellipsis, trailing far behind human performance.

Introduction

Visual dialogue requires "grounding"—the ability to link natural language to visual perception and shared context. However, existing datasets often suffer from two major flaws: undesirable biases that allow models to use statistical shortcuts, and coarse evaluation metrics that fail to explain failure modes. This research addresses these issues by focusing on spatial expressions, which serve as the backbone for referring in visual contexts. The authors provide a framework to evaluate models at a granular, structural level, diagnosing specific linguistic weaknesses.

Methodology and Data

The researchers utilize the OneCommon Corpus, a collaborative task where participants identify dots in partially overlapping views. This dataset is "bias-minimal" because visual attributes are generated from a uniform distribution, forcing models to rely on linguistic cues.

To analyze these dialogues, a three-step annotation schema was developed:

  1. Spatial Expression Detection: Identifying spans describing attributes (color, size) or relations (direction, proximity).
  2. Argument Identification: Treating spatial expressions as predicates and identifying their arguments. This includes handling argument ellipsis, where subjects are implied from previous turns.
  3. Canonicalization: Mapping natural language (e.g., "above," "higher than") into 24 canonical geometric and attribute relations.
Figure 1: Visualization of the OneCommon task and the proposed spatial expression annotation schema.

On the modeling side, the authors introduced NUMREF, which improves upon baseline reference models by adding a dedicated module to predict the number of referents before selection. This helps the model respect numerical constraints mentioned in the dialogue.

Figure 2: Architecture of the NUMREF model, highlighting the Number Prediction and Reference Resolution modules.

Results and Discussion

The analysis reveals a striking "What vs. Where" gap. Drawing on cognitive linguistics, the authors found that models are adept at recognizing "what" an object is (entity-level attributes like color) but fail at "where" it is (inter-entity relations).

Key findings include:

  • The Difficulty of Ellipsis: Models perform significantly worse when the subject of a spatial expression is elided and must be retrieved from a previous turn (57.14% success vs. 92.86% for humans).
  • Modification Sensitivity: Models are better at capturing "strong" modifications (e.g., "exactly horizontal") than nuanced ones (e.g., "slightly to the right").
  • Model Improvement: The NUMREF model significantly outperformed the baseline, jumping from 33.15% to 54.94% in exact match accuracy. However, a massive gap remains compared to the 86.90% human baseline.

Conclusion

This study provides a much-needed diagnostic lens for visual dialogue. By isolating spatial relations, the authors demonstrate that current models lack the structural understanding necessary for true grounding. Future work should focus on applying these explicit semantic constraints during training and developing better architectures for resolving ellipsis in complex visual scenes.

Summary updated

BibTeX
@inproceedings{udagawa2020linguistic,
  author = {Takuma Udagawa and Takato Yamazaki and Akiko Aizawa},
  title = {{A Linguistic Analysis of Visually Grounded Dialogues Based on Spatial Expressions}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.67},
}
Figure