Abstract
In the contemporary world, recording videos can be done quickly and easily. The quantity and availability of videos have continued to increase, therefore, an effective video retrieval method has also become important. To retrieve a target video from a large collection of videos, a video retrieval system needs to obtain appropriate queries from a user. Given a sentence query, there are many similar videos related to the query. The video retrieval system requires more information in addition to the sentence to distinguish the target video from others. If the system actively collects more information on the target video, we can perform video retrieval effectively. Thus, we propose a system to retrieve videos by asking questions about the content of the videos, and leveraging the user's responses to the questions and the dialog history. Additionally, we confirmed the usefulness of the proposed system through experiments using the dataset called AVSD which includes videos and dialogs about the videos.
Introduction
Searching for a specific video in a massive personal library, such as a smartphone's camera roll, is surprisingly difficult. A simple query like "a man reading a book" might return dozens of similar-looking clips. Standard web-based retrieval relies on metadata like tags or view counts, which are often absent in private video collections. This creates a "query gap": users frequently cannot remember enough specific details to write a perfect, discriminative query on the first attempt.
To bridge this gap, we propose an interactive agent that engages the user in a multi-turn dialog. By asking clarifying questions (e.g., "Is he sitting?" or "What room is he in?"), the system helps distinguish the target video from "distractor" videos that are semantically similar.
Proposed System Architecture
The proposed system consists of three main components working in an iterative loop to refine search results based on conversation history:
- History Encoder: To understand the context of the conversation, the model uses a hierarchical LSTM. It first encodes individual words into sentences, and then encodes the sequence of sentences (the dialog history) into a single "history vector." This allows the model to maintain context over multiple turns and "remember" previous interactions.
- Feature Embedding: This module maps both the history vector and the video features into a shared "joint embedding space." The system utilizes VSE++ (Visual-Semantic Embeddings) with hard-negative mining. By focusing on "hard negatives"—videos that are very similar to the target but incorrect—the model learns to identify fine-grained differences.
- Question Decoder: An LSTM-based module that takes the current dialog state and generates the next natural language question to ask the user, driving the interactive process forward.
Experimental Setup and Results
The researchers utilized the AVSD (Audio-Visual Scene-Aware Dialog) dataset, which contains nearly 8,000 training samples where two humans held a 10-round dialog to identify a specific video. For video representation, the system uses a multimodal approach combining motion (I3D), static appearance (ResNet152), and audio (VGGish) features.
Our results demonstrate that retrieval performance improves monotonically as the dialog progresses. Each round of Q&A acts as a filter, successfully pruning the candidate set. The proposed model significantly outperformed several baselines across all metrics:
- Recall@10: The proposed model achieved 24.4%, compared to just 5.1% for a standard LSTM baseline.
- Mean Rank (MeanR): The proposed model's average rank for the target video was 109 (out of 1,000), whereas the LSTM baseline sat at a distant 354.
- Feature Importance: Motion features (I3D) contributed more to retrieval accuracy than static features, suggesting that actions are more discriminative than settings in similar-looking scenes.
Conclusion and Future Work
This work highlights the power of dialog in video retrieval. By using a hierarchical encoder and hard-negative mining, the system effectively bridges the query gap. While highly effective, the current agent sometimes generates redundant questions. Future work will focus on developing "active" questioning strategies where the agent analyzes the current top-ranked candidates to decide which question would most effectively break the tie between them.
Summary updated