Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations

Wei Pang, Ruixue Duan, Jinfu Yang, Ning Li

TL;DR

The Multi-round Dialogue State Tracking (MDST) model moves away from treating visual dialog history as a flat text sequence, instead treating it as an evolving "state." By iteratively aligning image objects with textual entities and updating the dialogue state round-by-round, MDST achieves new state-of-the-art results on the VisDial v1.0 benchmark, producing more consistent and human-like answers.

What Problem Does This Paper Address?

Visual Dialog (VD) is a complex AI challenge: an agent must answer a series of questions about an image, requiring it to understand both the visual scene and the context of previous conversation rounds.

Most existing models take a "flat" approach—they simply concatenate the entire dialog history into one long string of text. The authors argue this is inefficient. It ignores the inherent "flow" of a conversation and fails to explicitly track how the state of the dialogue changes as new information is revealed. For example, if a user asks "What color is it?", the model needs to know what "it" refers to based on a specific previous round, rather than just searching through a giant block of text.

Methods

The authors introduce MDST, which treats the dialogue as a dynamic system. The core innovation is defining the dialogue state as a 2-tuple of vision and language representations:

  1. Vision States: Representations of specific objects in the image (fixed throughout the dialog).
  2. Language States: Representations of "dialogue entities" (updated every round).

The framework consists of four main modules:

  • QEncoder: A standard Transformer that encodes the current question and the initial image caption.
  • Question Grounding on Dialogue State (QGDS): This is the "brain" of the model. It uses a switching probability to decide whether the current question is more related to the visual objects or the previous dialogue history. It then aligns question words with both vision and language states to create a context-aware representation.
  • Answer Decoder (ADer): An autoregressive Transformer that generates a free-form natural language answer.
  • Postdiction on Dialogue State (PDS): After an answer is generated, this module "updates" the language state. It fuses the new question-answer pair into the existing dialogue state, preparing the model for the next round of conversation.

Datasets

The model was evaluated on VisDial v1.0, the standard large-scale dataset for visual dialogue. It contains 1 round of captioning followed by 10 rounds of question-answering for each image. The authors focused primarily on the generative setting, where the model must write its own answer rather than picking from a list.

Key Insights and Ideas

  • Iterative > Flat: Explicitly updating the dialogue state round-by-round is significantly more effective than treating history as a static text input.
  • The "Switching" Mechanism: Not every question requires looking at the history, and not every question requires looking at the image. The model’s ability to dynamically weight these two sources (via the switching probability $\phi$) is crucial for accuracy.
  • Object-Entity Alignment: By forcing the model to align visual objects with textual entities, it becomes much better at resolving coreferences (e.g., knowing that "the dog" in the text refers to a specific cluster of pixels in the image).
  • Measuring Consistency: The authors emphasize Joint Answer Accuracy (JACC). Instead of just checking if a single answer is right, JACC looks at whether the model can maintain a streak of correct answers across all 10 rounds.

Key Results

  • State-of-the-Art: MDST outperformed all previous models on the VisDial v1.0 generative setting, achieving an NDCG of 65.03 and an MRR of 53.49.
  • Consistency: The model achieved a JACC of 79.8%, meaning it gets roughly 8 out of 10 rounds correct in a typical conversation. This is a massive improvement over baseline models like LateFusion (53.4%).
  • Human-Like Quality: In human studies, MDST generated answers that were longer and more descriptive (average length of 3.57 words) compared to both the human ground truth (3.11) and previous models (1.81), without losing accuracy.

Limitations and Future Work

While MDST shows impressive results without massive pre-training, the authors note a few areas for growth:

  • Fixed Visual Objects: The vision states are extracted once at the start (using Faster-RCNN) and remain unchanged. Future versions might benefit from "dynamic" vision states that evolve as the conversation highlights different parts of the image.
  • Computational Complexity: Iteratively updating states for every round is more computationally intensive than simple concatenation, which might impact real-time deployment.
  • Scaling: The model was trained from scratch. Combining this state-tracking logic with massive pre-trained vision-language models (like CLIP or Flamingo) could potentially push performance even further.

Summary updated

Abstract

Visual Dialog (VD) is a task where an agent answers a series of image-related questions based on a multi-round dialog history. However, previous VD methods often treat the entire dialog history as a simple text input, disregarding the inherent conversational information flows at the round level. In this paper, we introduce Multi-round Dialogue State Tracking model (MDST), a framework that addresses this limitation by leveraging the dialogue state learned from dialog history to answer questions. MDST captures each round of dialog history, constructing internal dialogue state representations defined as 2-tuples of vision-language representations. These representations effectively ground the current question, enabling the generation of accurate answers. Experimental results on the VisDial v1.0 dataset demonstrate that MDST achieves a new state-of-the-art performance in generative setting. Furthermore, through a series of human studies, we validate the effectiveness of MDST in generating long, consistent, and human-like answers while consistently answering a series of questions correctly.
BibTeX
@misc{pang2024enhancing,
  author = {Wei Pang and Ruixue Duan and Jinfu Yang and Ning Li},
  title = {{Enhancing Visual Dialog State Tracking through Iterative Object-Entity Alignment in Multi-Round Conversations}},
  year = {2024},
  eprint = {2408.06725},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2408.06725},
}
Figure