Abstract
As online video content rapidly grows, the task of text-video retrieval (TVR) becomes increasingly important. A key challenge in TVR is the information asymmetry between video and text: videos are inherently richer in information, while their textual descriptions often capture only fragments of this complexity. This paper introduces a novel, data-centric framework to bridge this gap by enriching textual representations to better match the richness of video content. During training, videos are segmented into event-level clips and captioned to ensure comprehensive coverage. During retrieval, a large language model (LLM) generates semantically diverse queries to capture a broader range of possible matches. To enhance retrieval efficiency, we propose a query selection mechanism that identifies the most relevant and diverse queries, reducing computational cost while improving accuracy. Our method achieves state-of-the-art results across multiple benchmarks, demonstrating the power of data-centric approaches in addressing information asymmetry in TVR. This work paves the way for new research focused on leveraging data to improve cross-modal retrieval.
Introduction: The Information Asymmetry Problem
Text-Video Retrieval (TVR) aims to identify specific video clips from a database based on natural language queries. A fundamental hurdle in this field is information asymmetry. While a single video contains a massive amount of multi-layered information—including actions, objects, background details, and temporal transitions—the associated text queries or training captions are typically concise and only capture "fragments" of this complexity.
Because of this gap, retrieval models often learn to focus only on the specific snippets mentioned in the text, ignoring other potentially relevant visual data. This leads to biased training and brittle retrieval performance when a user's query does not perfectly align with the limited captions the model encountered during training.
Proposed Methodology
We propose a unified text enrichment framework that operates in two distinct stages: training and retrieval.
1. Training Phase: Event-Level Captioning
To ensure the model learns from the entire video, we move beyond global captions. We utilize the Kernel Temporal Segmentation (KTS) algorithm to decompose videos into "atomic scenes"—semantically consistent clips. A pre-trained Vision-Language Model (BLIP-2) then generates captions for the middle frame of every segment. This creates a "complete" textual representation that covers all significant moments within the video.
2. Retrieval Phase: Query Diversification and Selection
During inference, the system expands the user's initial query to capture a broader semantic range:
- LLM Expansion: GPT-4 generates 10 semantically diverse variations of the user's query (e.g., "a person cooking" vs. "someone preparing a meal in a kitchen").
- Farthest Query Sampling (FQS): To mitigate computational overhead and noise, FQS selects a small subset (typically $k=2$) of queries that are relevant to the original yet semantically distinct from each other.
- Aggregation: Results from these selected queries are combined using a majority voting strategy to determine the final video ranking.
Experimental Evaluation and Key Insights
The framework was evaluated on four standard benchmarks: MSR-VTT, MSVD, LSMDC, and VATEX. Our method achieved new state-of-the-art (SOTA) performance across all four, notably reaching a Rank-1 (R@1) of 58.9% on MSR-VTT.
The "Oracle Query" Revelation
A key insight from our study is the impact of the "Oracle Query" — the best possible query chosen from a set of generated variations. On MSR-VTT, replacing the original query with the Oracle version boosted R@1 from 46.7% to 61.4%. This suggests that the necessary information for retrieval is already present in the video; the challenge lies in how we "ask" for it.
Data-Centric vs. Model-Centric Approaches
While much research focuses on designing complex neural architectures, this work demonstrates that improving the quality and density of textual data is a highly effective lever for performance. Furthermore, the success of the FQS mechanism shows that semantic diversity in queries is more valuable than sheer quantity.
Conclusion
This paper presents a data-centric approach to bridging the information gap in text-video retrieval. By enriching text at both the training and retrieval stages, we significantly improve the model's ability to match complex visual content with natural language. Future work will focus on reducing inference latency and mitigating LLM hallucinations to further refine this framework.
Summary updated