TL;DR
This paper tackles the fundamental problem of "information asymmetry" in text-video retrieval—where dense, multi-layered videos are matched against sparse text descriptions—by introducing a novel data-centric text enrichment framework. By automatically generating event-level captions during training and using Large Language Models (LLMs) to diversify queries during retrieval, the proposed method achieves state-of-the-art performance across multiple benchmarks without requiring any architectural changes to the underlying models.
What Problem Does This Paper Address?
Text-video retrieval (TVR) systems typically project both text and video into a shared latent space to calculate their similarity. However, this approach assumes a symmetrical relationship between the two modalities that rarely exists in reality. In practice, videos are rich in visual scenes, actions, and background details, while their associated text queries or captions are highly concise and only capture fragments of this complexity.
This imbalance is known as information asymmetry. During training, it causes retrieval models to over-index on specific parts of a video while ignoring other potentially relevant aspects. During inference, if a user provides a query that fails to capture the exact semantics used during training, retrieval accuracy plummets. Instead of proposing a complex new model architecture, this paper addresses the issue from a data-centric perspective: how can we enrich textual representations to match the inherent richness of video content?
Methods
The authors introduce a unified text enrichment framework that operates during both the training and retrieval phases:
- Training Phase (Video Captioning Enrichment): To ensure full video coverage, videos are first segmented into non-overlapping, semantically consistent "atomic" scenes using the Kernel Temporal Segmentation (KTS) algorithm. A pre-trained Vision-Language Model (VLM) then generates a caption for the middle frame of each scene. This creates a comprehensive set of text descriptions covering the entire video timeline.
- Retrieval Phase (Query Diversification): When a user inputs a single sparse query, an LLM (such as GPT-4) is prompted to rewrite and expand it into multiple semantically diverse queries, capturing different potential visual perspectives and descriptions of the same scene.
- Query Selection via Farthest Query Sampling (FQS): Evaluating a large number of generated queries significantly increases computational overhead and can introduce noise from LLM hallucinations. To counter this, the authors introduce FQS, an algorithm that iteratively selects a small subset of queries ($k=2$) that maximize semantic diversity while keeping the original user query as a fixed anchor.
- Aggregation: The retrieval results from the selected queries are independently computed and then aggregated using a majority voting strategy to determine the final video ranking.
Datasets
The framework was evaluated on four standard text-video retrieval benchmarks:
- MSR-VTT: 10,000 video clips, each paired with 20 natural language captions.
- MSVD: 1,970 videos with approximately 40 English descriptions per video.
- LSMDC: 118,081 video clips from 202 movies, each with a single corresponding caption.
- VATEX: Approximately 35,000 video clips containing linguistically rich, multilingual annotations (English and Chinese).
Key Insights and Ideas
- The Power of the "Oracle Query": To find the upper bound of query expansion, the authors tested an "Oracle" setup where they selected the single best-performing query from the LLM-generated pool using ground-truth data. This simple swap boosted Rank-1 accuracy on MSR-VTT from 46.7% to a staggering 61.4%, proving that LLM expansions contain highly effective retrieval cues if properly selected.
- More Text is Not Always Better: Simply concatenating all generated queries into a single long description or average-pooling their embeddings actually degraded performance. Text encoders struggle to process overly long, dense descriptions, making independent query processing and rank aggregation (majority voting) essential.
- Strategic Pruning Beats Exhaustive Search: Using all 10 LLM-generated queries introduces noise and computational drag. Selecting just two maximally diverse queries via the FQS algorithm ($k=2$) actually outperformed using all 10 queries, demonstrating that diversity matters more than quantity.
Key Results
The framework achieved state-of-the-art results across all four benchmarks using standard backbones like CLIP-ViT-B/32 and ViT-B/16:
- MSR-VTT: Boosted Rank-1 accuracy from 46.9% (X-Pool baseline) to 52.1% (and up to 54.1% when paired with Dual Softmax Loss post-processing).
- MSVD & VATEX: Achieved top-tier Rank-1 accuracies of 51.1% and 63.3%, respectively.
- LSMDC: Reached a Rank-1 accuracy of 28.3%, outperforming specialized model-centric architectures.
- Architecture Agnostic: The retrieval-phase text enrichment proved highly effective as a training-free plug-in tool, consistently boosting the performance of other popular models like CLIP4Clip and CLIP-VIP.
Limitations and Future Work
- Hallucination and Noise: Relying on generative VLMs and LLMs inevitably introduces some hallucinated content, which can occasionally misguide the retrieval process.
- Prompt Engineering: The authors utilized a straightforward prompt for query generation; they note that exploring advanced prompting techniques like in-context learning could further improve query quality.
- LLM Inference Overhead: Although the FQS mechanism successfully minimizes downstream cross-modal similarity computations, the initial cost of calling an LLM at inference time remains a bottleneck. Future work will focus on optimizing this step using smaller, localized language models.
Summary updated