GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval

Zechen Bai, Tianjun Xiao, Tong He, Pichao Wang, Zheng Zhang, Thomas Brox, Mike Zheng Shou

Abstract

In the rapidly expanding domain of web video content, the task of text-video retrieval has become increasingly critical, bridging the semantic gap between textual queries and video data. This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video, enhancing the effectiveness of text-video retrieval systems. Unlike traditional model-centric methods that focus on designing intricate cross-modal interaction mechanisms, GQE aims to expand the text queries associated with videos both during training and testing phases. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions, effectively bridging the data imbalance gap. Furthermore, during retrieval, GQE utilizes Large Language Models (LLM) to generate a diverse set of queries and a query selection module to filter these queries based on relevance and diversity, thus optimizing retrieval performance while reducing computational overhead. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX, demonstrating the effectiveness of addressing text-video retrieval from a data-centric perspective.

Introduction: The Information Asymmetry Challenge

Text-video retrieval (TVR) aims to find the most relevant video clip given a natural language query. A fundamental challenge in this field is information asymmetry: videos are high-density data sources containing a multitude of actions, objects, and temporal dynamics, whereas the associated text descriptions are often sparse and "information-poor." For instance, a video showing a complex sequence of events might be labeled with a single, generic sentence.

Illustration of information asymmetry where textual queries often fail to capture all events present in a video.

This imbalance forces models to learn from incomplete signals, leading to poor performance when user queries focus on details omitted during training. While previous research has focused on complex architectural designs to bridge this gap, this paper proposes a data-centric alternative: Generalized Query Expansion (GQE).

Methodology: The GQE Framework

The GQE framework addresses information asymmetry by enriching textual data at both the training and retrieval stages. This approach allows existing models to benefit from denser semantic signals without requiring fundamental changes to their architecture.

The GQE framework consists of three main components: text enrichment during training, the core model training, and text enrichment during retrieval.

1. Text Enrichment in Training

To provide a denser learning signal, GQE employs Kernel Temporal Segmentation (KTS) to break long videos into "atomic" event-level clips. A Vision-Language Model (VLM) then generates zero-shot captions for each segment. This process ensures that every semantically significant moment in a video is represented in the text, allowing the model to associate diverse visual features with corresponding textual descriptions.

2. Text Enrichment in Retrieval

During the inference phase, a user's query may not perfectly match the learned video representations. GQE uses a Large Language Model (LLM) to expand a single user query into multiple semantically diverse versions. To maintain efficiency, the authors introduced Farthest Query Sampling (FQS), which selects a small subset of these generated queries that maximize diversity while remaining relevant. This "query diversification" increases the probability of a successful match between the user's intent and the video content.

Experimental Results

The researchers evaluated GQE across four standard benchmarks: MSR-VTT, MSVD, LSMDC, and VATEX.

  • State-of-the-Art Performance: GQE achieved a Rank@1 (R@1) of 58.9% on MSR-VTT using a ViT-B/32 backbone, significantly outperforming previous model-centric approaches.
  • The Oracle Query Insight: An "Oracle" experiment—picking the best possible query from the LLM-generated list—showed a potential jump to 61.4% R@1 on MSR-VTT. This confirms that the necessary information is present in the video; the bottleneck is the textual interface.
  • Efficiency and Robustness: Using FQS with just two queries ($k=2$) was found to be more effective than using ten redundant ones, proving that diversity is more critical than sheer volume. Additionally, the LLM-based expansion naturally corrected user typos and semantic ambiguities.

Conclusion

GQE demonstrates that the "information asymmetry" in text-video retrieval is best addressed through a data-centric lens. By enriching text during both training (via event-aware captioning) and retrieval (via LLM-powered expansion), GQE unlocks the latent potential of existing vision-language models without requiring complex new architectures. Future work will explore the use of smaller, on-device LLMs to reduce latency and further optimize prompt engineering for even more precise query expansion.

Summary updated

Abstract

In the rapidly expanding domain of web video content, the task of text-video retrieval has become increasingly critical, bridging the semantic gap between textual queries and video data. This paper introduces a novel data-centric approach, Generalized Query Expansion (GQE), to address the inherent information imbalance between text and video, enhancing the effectiveness of text-video retrieval systems. Unlike traditional model-centric methods that focus on designing intricate cross-modal interaction mechanisms, GQE aims to expand the text queries associated with videos both during training and testing phases. By adaptively segmenting videos into short clips and employing zero-shot captioning, GQE enriches the training dataset with comprehensive scene descriptions, effectively bridging the data imbalance gap. Furthermore, during retrieval, GQE utilizes Large Language Models (LLM) to generate a diverse set of queries and a query selection module to filter these queries based on relevance and diversity, thus optimizing retrieval performance while reducing computational overhead. Our contributions include a detailed examination of the information imbalance challenge, a novel approach to query expansion in video-text datasets, and the introduction of a query selection strategy that enhances retrieval accuracy without increasing computational costs. GQE achieves state-of-the-art performance on several benchmarks, including MSR-VTT, MSVD, LSMDC, and VATEX, demonstrating the effectiveness of addressing text-video retrieval from a data-centric perspective.
BibTeX
@inproceedings{bai2025gqe,
  author = {Zechen Bai and Tianjun Xiao and Tong He and Pichao Wang and Zheng Zhang and Thomas Brox and Mike Zheng Shou},
  title = {{GQE: Generalized Query Expansion for Enhanced Text-Video Retrieval}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=Tn6lrFbiP4},
}
Figure