Saying the Unseen: Video Descriptions via Dialog Agents

Ye Zhu, Yu Wu, Yi Yang, Yan Yan

Abstract

Current vision and language tasks typically rely on complete visual data, which can be problematic in scenarios with restricted views or privacy concerns. We introduce a novel task that describes a video using natural language dialog between two agents as a supplementary source for incomplete visual data. One agent, Q-BOT, receives only two semantic segmented frames and must ask questions to describe the unseen video. The other agent, A-BOT, has full access and answers these questions. Using unified QA-Cooperative networks, we demonstrate that natural language dialog effectively supplements missing visual information while maintaining privacy and transparency.

Introduction

Most current vision-language models—like those for video captioning or visual question answering (VQA)—rely on full access to raw video data. While effective, this poses significant privacy risks, as raw footage often contains sensitive biometric information like recognizable faces or voices. Furthermore, in many real-world scenarios, such as fixed-view security cameras or smart home sensors, visual data is naturally incomplete.

This paper introduces the concept of implicit vision: a setup where visual information is intentionally restricted to protect privacy. The challenge is: how can an AI system still generate a meaningful description of a scene if it can't "see" the whole thing? The solution proposed is to use natural language dialog as a supplementary information source, allowing a "blind" agent to query a "seeing" agent to fill in the blanks.

The Cooperative Dialog Task

The task is structured as a cooperative game between two agents:

  1. Q-BOT (Questioner): Receives only two semantically segmented frames (the first and last) of a video. These frames show object categories but hide identities and fine details. Q-BOT must ask up to 10 questions to understand what happened in between.
  2. A-BOT (Answerer): Has full access to the video, audio, and original human-written descriptions. It answers Q-BOT’s questions.
The three-phase task setup: Input, Internal Dialog, and Video Description.

The researchers developed QA-Cooperative networks to handle this interaction. They explored two experimental settings: a Generative setting where agents generate text freely, and a Discriminative setting where agents select the best response from candidates. The discriminative approach was found to be more interpretable and less prone to the repetitive "looping" often seen in generative models.

Data and Privacy-Preserving Vision

The study primarily uses the Audio-Visual Scene-Aware Dialog (AVSD) dataset. To implement the "implicit vision" constraint, the authors modified the input for Q-BOT by passing raw frames through a pre-trained semantic segmentation model (PSPNet). This ensures that no biometric data is visible to the questioning agent, only the semantic "sketches" of the environment.

Q-BOT receives segmented 'sketches' to preserve privacy while maintaining spatial context.

Key Insights and Results

  • Dialog as a Transparent Bridge: Unlike obscure high-dimensional feature vectors, natural language dialog is "human-readable." This makes the information transfer between agents transparent and auditable.
  • The Power of Selection: The discriminative setting significantly outperformed the generative one. By picking from candidates, the agents maintained better logic and provided more diverse, informative details about the unseen video.
  • Privacy without Blindness: Even with heavily redacted visual input, Q-BOT was able to generate descriptions that closely matched those produced by models with full visual access, provided the dialog was high-quality. The proposed QA-Cooperative model (Discriminative) achieved a CIDEr score of 27.1, a significant jump over the basic baseline without dialog (14.2).
  • Causality in Dialog: The order of questions matters. Shuffling the dialog history or reducing the number of rounds significantly degraded the final description, proving that Q-BOT builds a mental model of the scene over time.

Conclusion

This research demonstrates that natural language can serve as a secure and effective medium to bridge the "knowledge gap" in incomplete visual data. By intentionally limiting visual input and using dialog to fill the gaps, the authors provide a path toward more secure and transparent AI systems. Future work will explore efficiency improvements, such as early-stopping mechanisms, and enhancing the descriptive capabilities of the expert agent.

Summary updated

Abstract

Current vision and language tasks usually take complete visual data (e.g., raw images or videos) as input, however, practical scenarios may often consist the situations where part of the visual information becomes inaccessible due to various reasons e.g., restricted view with fixed camera or intentional vision block for security concerns. As a step towards the more practical application scenarios, we introduce a novel task that aims to describe a video using the natural language dialog between two agents as a supplementary information source given incomplete visual data. Different from most existing vision-language tasks where AI systems have full access to images or video clips, which may reveal sensitive information such as recognizable human faces or voices, we intentionally limit the visual input for AI systems and seek a more secure and transparent information medium, i.e., the natural language dialog, to supplement the missing visual information. Specifically, one of the intelligent agents - Q-BOT - is given two semantic segmented frames from the beginning and the end of the video, as well as a finite number of opportunities to ask relevant natural language questions before describing the unseen video. A-BOT, the other agent who has access to the entire video, assists Q-BOT to accomplish the goal by answering the asked questions. We introduce two different experimental settings with either a generative (i.e., agents generate questions and answers freely) or a discriminative (i.e., agents select the questions and answers from candidates) internal dialog generation process. With the proposed unified QA-Cooperative networks, we experimentally demonstrate the knowledge transfer process between the two dialog agents and the effectiveness of using the natural language dialog as a supplement for incomplete implicit visions.
BibTeX
@article{zhu2022saying,
  author = {Ye Zhu and Yu Wu and Yi Yang and Yan Yan},
  title = {{Saying the Unseen: Video Descriptions via Dialog Agents}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2022},
  url = {https://openreview.net/forum?id=OWM51qPeCsZ},
}
Figure