Abstract
As the volume of personal video recordings continues to grow, retrieving a specific clip from a large, unlabeled collection becomes increasingly difficult. This research addresses the challenge of disambiguating similar videos by introducing an interactive dialog between the system and the user. We propose a system that retrieves target videos by asking clarifying questions and leveraging user responses. Experiments on the Audio-Visual Scene-Aware Dialog (AVSD) dataset demonstrate that retrieval accuracy significantly improves as the conversation progresses, effectively narrowing down candidates through multi-round interaction.
Introduction: Finding the Needle in the Video Haystack
Searching for a specific video in a massive personal collection—like a phone’s camera roll or a life-log database—is notoriously difficult. Unlike web videos, these clips lack tags or SEO-friendly descriptions. A simple query like "a man reading a book" might return dozens of similar results.
Researchers from the University of Tokyo and RIKEN propose a solution: Interactive Video Retrieval with Dialog. Instead of a one-shot search, the system acts as an agent that "talks" to the user, asking clarifying questions to narrow down the candidates. This approach focuses on disambiguation, helping the system pick the "Ground Truth" (GT) video out of a crowd of look-alikes.
System Architecture
The proposed system utilizes a modular architecture designed to handle multi-round conversations and complex video features.
The architecture consists of three primary components:
- History Encoder: Uses a Hierarchical Recurrent Encoder (LSTMs) to transform the dialog history—including the initial caption and subsequent Q&A pairs—into a single "history vector." The model prioritizes the initial query while remaining flexible enough to integrate new details.
- Feature Embedding: Maps the history vector and video features into a joint embedding space. A ranking loss function with "hard negatives" is used to help the model distinguish between very similar clips.
- Question Decoder: An LSTM-based module that generates the next logical question based on the current history, ensuring the agent actively seeks missing information.
For video representation, the system combines spatial features (ResNet152), spatio-temporal motion features (I3D), and audio features (VGGish).
Experimental Evaluation
The system was evaluated using the AVSD (Audio-Visual Scene-Aware Dialog) dataset, which contains videos of daily activities paired with ten rounds of descriptive questions and answers.
Quantitative Performance
The proposed model outperformed several baselines. A key finding was the importance of motion: spatio-temporal features (I3D) significantly outperformed static frame features because many videos are distinguished by actions (e.g., walking vs. standing) rather than just objects.
As shown in the performance trends, the system gets "smarter" the longer the interaction lasts. On the AVSD dataset, the model achieved a Recall@10 of 22.1% and a Mean Rank of 119 after 10 rounds of dialog, compared to 16.8% Recall@10 for a basic version of the model.
Human-in-the-Loop Validation
A user study via Amazon Mechanical Turk confirmed the system's effectiveness with real humans. In these trials, the Mean Rank of the target video improved steadily as humans answered the agent's questions. For example, an initial search for a person in a "doorway" might be ambiguous, but after the agent asks about the room and the user replies "kitchen," the system successfully prioritizes the correct video.
Limitations and Future Work
While promising, the current model has areas for improvement. Video features are currently pooled in the time direction, which may lose granular temporal details; future work could explore attention mechanisms for better feature handling. Additionally, while the system generates appropriate questions, optimizing the strategy of questioning—asking the most informative questions first—could further minimize the number of rounds required for successful retrieval.
Summary updated