Saying the Unseen: Video Descriptions via Dialog Agents

Ye Zhu, Yu Wu, Yi Yang, Yan Yan

TL;DR

This paper introduces a novel multi-agent task where an AI agent (Q-BOT) must describe a video it hasn't fully seen by engaging in a natural language dialog with an "expert" agent (A-BOT). By intentionally limiting visual input to a few privacy-preserving segmented frames, the researchers demonstrate that natural language can serve as a secure and effective medium to bridge the "knowledge gap" in incomplete visual data.

What Problem Does This Paper Address?

Most current vision-language models—like those for video captioning or visual question answering (VQA)—rely on full access to raw video data. While effective, this poses significant privacy risks, as raw footage often contains sensitive biometric information like recognizable faces or voices. Furthermore, in many real-world scenarios (e.g., fixed-view security cameras or smart home sensors), visual data is naturally incomplete.

The authors propose the concept of implicit vision: a setup where visual information is intentionally restricted to protect privacy. The challenge is: how can an AI system still generate a meaningful description of a scene if it can't "see" the whole thing? The solution proposed here is to use natural language dialog as a supplementary information source, allowing a "blind" agent to query a "seeing" agent to fill in the blanks.

Methods

The task is structured as a cooperative game between two agents:

  1. Q-BOT (Questioner): Receives only two semantically segmented frames (the first and last) of a video. These frames show object categories but hide identities and fine details. Q-BOT must ask up to 10 questions to understand what happened in between.
  2. A-BOT (Answerer): Has full access to the video, audio, and original human-written descriptions. It answers Q-BOT’s questions.
The three-phase task setup: Input, Internal Dialog, and Video Description.

The researchers developed QA-Cooperative networks to handle this interaction. They explored two experimental settings:

  • Generative: Agents generate questions and answers freely word-by-word.
  • Discriminative: Agents select the best question or answer from a pool of candidates. This was found to be more interpretable and less prone to the repetitive "looping" often seen in generative models.

The architecture (shown in Figure 3) utilizes Multi-Modal (MM) attention to fuse visual, audio, and textual history, along with a dynamic history update mechanism to ensure Q-BOT integrates new information from each dialog round.

The detailed architecture of the QA-Cooperative network, highlighting the different information flows for Q-BOT and A-BOT.

Datasets

The study primarily uses the Audio-Visual Scene-Aware Dialog (AVSD) dataset. Originally designed for VQA, the authors repurposed it for their "Saying the Unseen" task. They modified the input for Q-BOT by passing raw frames through a pre-trained semantic segmentation model (PSPNet), ensuring that no biometric data is visible to the questioning agent.

Comparison of inputs: Q-BOT sees segmented "sketches," while A-BOT sees the full raw video.

Key Insights and Ideas

  • Dialog as a Transparent Bridge: Unlike obscure high-dimensional feature vectors, natural language dialog is "human-readable." This makes the information transfer between agents transparent and auditable.
  • The Power of Selection: The discriminative setting significantly outperformed the generative one. By picking from candidates, the agents maintained better logic and provided more diverse, informative details about the unseen video.
  • Privacy doesn't mean "Blindness": Even with heavily redacted visual input (segmented frames), Q-BOT was able to generate descriptions that closely matched those produced by models with full visual access, provided the dialog was high-quality.
  • Causality in Dialog: The order of questions matters. Shuffling the dialog history or reducing the number of rounds significantly degraded the final description, proving that Q-BOT actually "learns" and builds a mental model of the scene over time.

Key Results

The proposed QA-Cooperative model (Discriminative) achieved a CIDEr score of 27.1, a significant jump over the basic baseline without dialog (14.2) and the previous generative state-of-the-art (22.9).

Qualitative results (Figure 4) show that Q-BOT can correctly identify complex actions—like a person vacuuming or drinking coffee—that are completely absent from its initial two-frame input, solely by asking the right questions.

Qualitative examples showing how Q-BOT's descriptions become more detailed and accurate after the internal dialog.

Limitations and Future Work

While promising, the authors note several areas for improvement:

  • Efficiency: Q-BOT currently uses all 10 dialog rounds. Future versions could implement an "early stopping" mechanism if the agent feels it has enough information, making the interaction more efficient.
  • A-BOT Performance: There is still a gap between A-BOT’s "expert" descriptions and Q-BOT’s "reconstructed" ones. Improving A-BOT’s ability to provide more descriptive, less biased answers is a key goal.
  • Task Specificity: Instead of general summaries, future work could task Q-BOT with generating specific outputs like scene graphs or action sequences based on the dialog.

Summary updated

Abstract

Current vision and language tasks usually take complete visual data (e.g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e.g., restricted view with fixed camera or intentional vision block for security concerns. As a step towards the more practical application scenarios, we introduce a novel task that aims to describe a video using the natural language dialog between two agents as a supplementary information source given incomplete visual data. Different from most existing vision-language tasks where AI systems have full access to images or video clips, which may reveal sensitive information such as recognizable human faces or voices, we intentionally limit the visual input for AI systems and seek a more secure and transparent information medium, i.e., the natural language dialog, to supplement the missing visual information. Specifically, one of the intelligent agents - <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q-BOT</i> - is given two semantic segmented frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">A-BOT</i> , the other agent who has access to the entire video, assists <italic xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">Q-BOT</i> to accomplish the goal by answering the asked questions. We introduce two different experimental settings with either a generative (i.e., agents generate questions and answers freely) or a discriminative (i.e., agents select the questions and answers from candidates) internal dialog generation process. With the proposed unified QA-Cooperative networks, we experimentally demonstrate the knowledge transfer process between the two dialog agents and the effectiveness of using the natural language dialog as a supplement for incomplete implicit visions.
BibTeX
@article{zhu2022saying,
  author = {Ye Zhu and Yu Wu and Yi Yang and Yan Yan},
  title = {{Saying the Unseen: Video Descriptions via Dialog Agents}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=OWM51qPeCsZ},
}

Loading related content...

Figure