Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge

Brielen Madureira, David SchlangenACL 2022 · DOI

Introduction

In human conversation, participants maintain a "mental scoreboard" to track mutual knowledge—facts established as shared versus those that remain private to one individual. For instance, if an Answerer informs a Questioner that "there is a dog outside," that proposition moves from private to shared knowledge. If the dialogue fails to update this state correctly, subsequent references (e.g., "What color is the dog?") become incoherent.

This paper investigates whether state-of-the-art visual dialogue models, typically trained end-to-end, develop internal representations of this evolving knowledge state. By applying linguistic theories of grounding, the authors evaluate whether neural models perform "scorekeeping" or merely rely on surface-level pattern matching.

Methodology: Probing the Scoreboard

The authors propose a diagnostic probing task to examine the hidden representations of pretrained visual dialogue encoders. The core of this approach is a probing classifier trained on top of frozen weights from models like those used in the VisDial challenge.

Overview of the probing framework. A frozen dialogue encoder's representation is concatenated with a proposition embedding to predict its status on the 'scoreboard' across four classes: True/False and Private/Shared.

The Probing Task

The classifier (a small Multi-Layer Perceptron) takes a dialogue state representation and a specific proposition embedding (generated via Sentence-BERT) to predict the proposition's status at a specific turn. Status is categorized into four classes:

  1. True to Answerer / Private
  2. True to Answerer / Shared
  3. False to Answerer / Private
  4. False to Answerer / Shared

To ensure the probe measures the dialogue representation rather than memorizing propositions, control tasks were implemented using random or null vectors as baselines.

Dataset and Proposition Generation

The study utilizes the VisDial v1.0 dataset. To generate the probing data, the authors developed 34 programmatic rules to transform Question-Answer pairs into declarative propositions (e.g., "Is the cat sleeping?" + "Yes" becomes "The cat is sleeping"). They applied coreference resolution to handle pronouns and balanced the dataset to prevent simple frequency biases from inflating accuracy.

Key Results and Analysis

The probing experiments reveal that while models encode some scorekeeping information, they fall short of human-level consistency.

  • Accuracy: Models achieved approximately 61-62% accuracy on the 4-class task (TFxPS) at turn 5, significantly outperforming random-vector controls (which scored ~35-37%) but trailing far behind the human baseline of ~92%.
  • Dialogue Saliency: Models are notably better at identifying the truth value of propositions once they have been mentioned in the dialogue (shared) than when they are still private. This suggests that the dialogue history provides a more salient signal than the raw visual information.
  • Incremental Inconsistency: A critical finding was the lack of stability in model predictions. In a perfect system, a fact's status should transition logically (e.g., from private to shared) and remain stable. However, models frequently exhibited "oscillations," where a fact might be predicted as shared at turn 5 but revert to private at turn 7.
Confusion matrix for the 4-class classification task. The results show that models are relatively proficient at distinguishing truth values (T vs F) but struggle more with the private/shared (S vs P) distinction.

Discussion and Limitations

The authors conclude that current visual dialogue models do not naturally develop robust, human-like scorekeeping capabilities. Only 44.5% of propositions maintained stable truth-value predictions throughout a dialogue, and only 38.2% of knowledge shifts occurred at the correct turn.

One primary limitation is the "passive" nature of the VisDial task; because agents are not required to coordinate to achieve a specific goal, there is little pressure for the models to maintain a precise internal state of shared knowledge. Future work should focus on "ecologically valid" datasets that necessitate consistent knowledge tracking for task success, potentially involving more complex transformer-based architectures and larger multimodal models.

Summary updated

BibTeX
@inproceedings{madureira2022can,
  author = {Brielen Madureira and David Schlangen},
  title = {{Can Visual Dialogue Models Do Scorekeeping? Exploring How Dialogue Representations Incrementally Encode Shared Knowledge}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.acl-short.73},
}
Figure