Multi-modal Dialogue State Tracking for Playing GuessWhich Game

Wei Pang, Ruixue Duan, Yang Jinfu, Ning LiLecture Notes in Computer Science 2024 · DOI · arXiv · OpenAlex

In the game of "GuessWhich," one player (the Questioner) tries to identify a secret image by asking another player (the Answerer) a series of questions. For AI, this is a grueling test of multi-modal reasoning. How do you ask a smart question about something you can't see?

A new paper titled "Multi-modal Dialogue State Tracking for Playing GuessWhich Game" proposes a shift in how we build these agents. Instead of just processing text or "cheating" by looking at random real images, the authors introduce a QBot that builds its own "mental imagery" to track the state of the conversation.

TL;DR

The authors propose a Dialogue State Tracking (DST) model for the GuessWhich game that maintains a dynamic "mental model" of a hidden image. By aligning textual entities with evolving visual representations, their agent achieves new state-of-the-art performance on the VisDial datasets, significantly outperforming models that rely solely on text or external image sampling.

What Problem Does This Paper Address?

In the GuessWhich game, the Questioner Bot (QBot) starts with only a brief caption. To find the correct image in a pool of thousands, it must ask strategic questions.

Current approaches generally fall into two camps:

  1. Non-visual QBots: These rely entirely on language models. They struggle to link words to actual visual concepts, often leading to repetitive or irrelevant questions.
  2. Real-image QBots: These sample a "likely" image from a database at each round to use as context. This is computationally expensive, "unnatural" (humans don't flip through a physical photo album to think), and introduces high variance that makes training unstable.

The challenge is creating a bot that can perform visually related reasoning internally, much like a human forms a "mental picture" while listening to a description.

Methods

The researchers introduced the Dialogue State Tracking (DST) model, which revolves around a cycle of two procedures:

  • State Tracking (STrack): As the dialogue progresses, the bot maintains a "Dialogue State" consisting of two parts: Word States (textual representations of entities like "strawberry") and Image States (mental visual concepts). Using a Gumbel-Softmax layer, the bot decides whether to Add a new entity to its mental model or Update an existing one with new info (e.g., "the strawberry is red").
  • Visual Reasoning on Dialogue State (VRDS): This module uses a novel Recursive Self-Reference Equation (R-SRE). It performs "three-hop reasoning" (words $\rightarrow$ words $\rightarrow$ images $\rightarrow$ words) to synthesize the current mental model into a context vector used to generate the next question.

To train the model, they used a combination of standard Cross-Entropy loss for language and a Progressive Loss (PL). The PL is particularly clever: it rewards the bot for making its mental representation more similar to the ground-truth image with every passing round.

Datasets

The model was evaluated on the three standard benchmarks for this field:

  • VisDial v0.5
  • VisDial v0.9
  • VisDial v1.0

These datasets contain tens of thousands of images and human-human dialogues, providing a rigorous testing ground for image retrieval and question diversity.

Key Insights and Ideas

  • Dual-Coding Theory: The model is inspired by the psychological theory that human cognition uses two interconnected systems: one for language and one for images. By forcing the bot to maintain an "Image State," the researchers successfully mimicked this human trait.
  • Dynamic State Growth: Unlike static models, this DST model grows its internal representation. If the dialogue mentions a "carpet," a new slot is created in the mental model. This prevents the "forgetting" or "confusion" common in long dialogues.
  • Avoiding Repetition: Because the bot tracks what it already "knows" in its mental state, it is much less likely to ask the same question twice—a common failure mode for earlier QBots.

Key Results

The DST model established a new State-of-the-Art (SOTA) across all metrics:

  • Image Guessing: On VisDial v1.0 (validation), it achieved a Percentile Mean Rank (PMR) of 99.60 and a Mean Rank of 17.52, significantly outperforming the previous best model (ReeQ), which sat at 99.13 and 18.05 respectively.
  • Efficiency: Compared to "Real-image" models that have to retrieve images from large pools (e.g., 9,628 candidates), the DST model's internal mental imagery is far more efficient and robust.
  • Qualitative Superiority: Case studies showed the bot asking specific, color-oriented questions (e.g., "What color is the bear?") much earlier in the conversation than baseline models, indicating a more focused search strategy.

Limitations and Future Work

While the model creates "mental imagery" in the form of high-dimensional feature vectors, it doesn't actually "draw" a picture. The authors note that visualizing the internal image state—actually rendering what the bot "sees" in its head—is a promising direction for future research. This would make the AI's reasoning process even more interpretable to humans. Additionally, exploring how this mental model scales with extremely long, complex dialogues remains an open question.

Summary updated

BibTeX
@inproceedings{pang2024multi,
  author = {Wei Pang and Ruixue Duan and Yang Jinfu and Ning Li},
  title = {{Multi-modal Dialogue State Tracking for Playing GuessWhich Game}},
  booktitle = {Lecture Notes in Computer Science},
  year = {2024},
  doi = {10.1007/978-981-99-8850-1\_45},
}

Loading related content...

Figure