Interactive Video Retrieval with Dialog

Sho Maeoki, Kohei Uehara, Tatsuya Harada

Abstract

As the volume of personal video recordings continues to grow, retrieving a specific clip from a large, unlabeled collection becomes increasingly difficult. This research addresses the challenge of disambiguating similar videos by introducing an interactive dialog between the system and the user. We propose a system that retrieves target videos by asking clarifying questions and leveraging user responses. Experiments on the Audio-Visual Scene-Aware Dialog (AVSD) dataset demonstrate that retrieval accuracy significantly improves as the conversation progresses, effectively narrowing down candidates through multi-round interaction.

Introduction: Finding the Needle in the Video Haystack

Searching for a specific video in a massive personal collection—like a phone’s camera roll or a life-log database—is notoriously difficult. Unlike web videos, these clips lack tags or SEO-friendly descriptions. A simple query like "a man reading a book" might return dozens of similar results.

Researchers from the University of Tokyo and RIKEN propose a solution: Interactive Video Retrieval with Dialog. Instead of a one-shot search, the system acts as an agent that "talks" to the user, asking clarifying questions to narrow down the candidates. This approach focuses on disambiguation, helping the system pick the "Ground Truth" (GT) video out of a crowd of look-alikes.

System Architecture

The proposed system utilizes a modular architecture designed to handle multi-round conversations and complex video features.

The modular architecture of the proposed interactive retrieval system, featuring the History Encoder, Question Decoder, and Feature Embedding modules.

The architecture consists of three primary components:

  1. History Encoder: Uses a Hierarchical Recurrent Encoder (LSTMs) to transform the dialog history—including the initial caption and subsequent Q&A pairs—into a single "history vector." The model prioritizes the initial query while remaining flexible enough to integrate new details.
  2. Feature Embedding: Maps the history vector and video features into a joint embedding space. A ranking loss function with "hard negatives" is used to help the model distinguish between very similar clips.
  3. Question Decoder: An LSTM-based module that generates the next logical question based on the current history, ensuring the agent actively seeks missing information.

For video representation, the system combines spatial features (ResNet152), spatio-temporal motion features (I3D), and audio features (VGGish).

Experimental Evaluation

The system was evaluated using the AVSD (Audio-Visual Scene-Aware Dialog) dataset, which contains videos of daily activities paired with ten rounds of descriptive questions and answers.

Quantitative Performance

The proposed model outperformed several baselines. A key finding was the importance of motion: spatio-temporal features (I3D) significantly outperformed static frame features because many videos are distinguished by actions (e.g., walking vs. standing) rather than just objects.

Recall@10 performance across 10 rounds of dialog. The proposed model shows a clear upward trend in retrieval accuracy as more information is gathered.

As shown in the performance trends, the system gets "smarter" the longer the interaction lasts. On the AVSD dataset, the model achieved a Recall@10 of 22.1% and a Mean Rank of 119 after 10 rounds of dialog, compared to 16.8% Recall@10 for a basic version of the model.

Human-in-the-Loop Validation

A user study via Amazon Mechanical Turk confirmed the system's effectiveness with real humans. In these trials, the Mean Rank of the target video improved steadily as humans answered the agent's questions. For example, an initial search for a person in a "doorway" might be ambiguous, but after the agent asks about the room and the user replies "kitchen," the system successfully prioritizes the correct video.

Limitations and Future Work

While promising, the current model has areas for improvement. Video features are currently pooled in the time direction, which may lose granular temporal details; future work could explore attention mechanisms for better feature handling. Additionally, while the system generates appropriate questions, optimizing the strategy of questioning—asking the most informative questions first—could further minimize the number of rounds required for successful retrieval.

Summary updated

Abstract

Now that everyone can easily record videos, the quantity of which is continuously increasing, research on methods for improved video retrieval is important in the contemporary world. In cases where target videos are to be identified within a large collection gathered by individuals, the appropriate information must be obtained to retrieve the correct video within a large number of similar items in the target database. The purpose of this research is to retrieve target videos in such cases by introducing an interaction, or a dialog, between the system and the user. We propose a system to retrieve videos by asking questions about the content of the videos and leveraging the user's responses to the questions. Additionally, we confirmed the usefulness of the proposed system through experiments using the dataset called AVSD which includes videos and dialogs about the videos.
BibTeX
@inproceedings{maeoki2020interactive,
  author = {Sho Maeoki and Kohei Uehara and Tatsuya Harada},
  title = {{Interactive Video Retrieval with Dialog}},
  booktitle = {Workshops @ CVPR},
  year = {2020},
  url = {https://openreview.net/forum?id=8Jxil0q3US5},
}
Figure