PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search

Jiaxin Wu, Chong-Wah Ngo, Xiao-Yong Wei, Qing LiarXiv 2024 · DOI · arXiv

Introduction

Ad-hoc Video Search (AVS) represents a significant challenge in the field of computer vision and information retrieval: the ability to locate specific video segments using arbitrary, natural language queries. While modern embedding models have made strides, the PolySmart and VIREO team identifies three persistent bottlenecks: out-of-vocabulary (OOV) terms that fall outside a model's training set, complex logical constraints (e.g., "and/or" operations), and nuanced spatial relationships (e.g., "inside" vs. "in front of").

To address these, the team developed a generation-augmented retrieval framework for the TRECVid 2024 AVS task. By leveraging generative AI to "re-imagine" and expand user queries across multiple modalities, they successfully bridged the gap between human intent and machine-searchable video content.

Methodology: Generation-Augmented Retrieval

The core of the proposed system is a plug-in module that augments the original user query through three distinct generative pathways, as illustrated in the system architecture below:

System architecture showing the flow from user query through T2T, T2I, and I2T modules to the search engine and final ensemble.

Multi-Modal Query Expansion

  1. Text-to-Text (T2T): Using Llama 3, the system rephrases queries to align with the search engine's internal "concept bank." This effectively translates OOV words into synonyms that the retrieval model can recognize.
  2. Text-to-Image (T2I): The query is processed by Stable Diffusion to generate synthetic images. This step is crucial for capturing spatial and logical constraints that are often lost in text-only embeddings.
  3. Image-to-Text (I2T): These synthetic images are then captioned using BLIP-2, providing a rich, descriptive context that offers a different linguistic perspective on the original query.

Finally, the system retrieves video rankings for all four versions (original + three augmented) and fuses them using a linear combination to produce the final result.

Visualizing Logic and Spatial Constraints

One of the most innovative aspects of this approach is using Text-to-Image generation to resolve complex spatial queries. While a text model might struggle to distinguish between "in front of" and "inside," a generative image model can visually represent these relationships, providing a concrete anchor for the search engine.

Examples of T2I generation handling complex spatial and logical queries, such as being inside a garage or wearing a specific colored scarf.

As shown in the figure above, the T2I model successfully visualizes logical "or" constraints (e.g., red or blue scarves) and spatial positioning, which helps the retrieval engine focus on the correct visual features.

Results and Performance

The team evaluated their approach on the V3C2 dataset using 30 complex queries. The results, measured in mean xinfAP (Extended Inferred Average Precision), demonstrate the clear advantage of the fusion approach.

Run ID Query Type xinfAP
Run 1 Original + T2T + T2I + I2T (Fusion) 0.294
Run 2 T2T query only 0.283
Run 3 I2T query only 0.277
Run 4 Original query only 0.277
Run Novelty T2I query only 0.216

The fusion model (Run 1) achieved the highest performance, outperforming the original query baseline by a significant margin. Interestingly, while T2I alone had the lowest individual score, its contribution to the final ensemble was vital for resolving the spatial and logical nuances that text-only models often miss.

Conclusion and Future Directions

The PolySmart and VIREO team's success at TRECVid 2024 highlights the potential of generative AI as a pre-processing step for complex retrieval tasks. By treating LLMs and diffusion models as "translators" and "visualizers," the system overcomes traditional OOV and spatial reasoning hurdles. Future work will focus on improving "query attention" in image-to-video retrieval to ensure specific attributes, such as color and quantity, are preserved more accurately throughout the pipeline.

Summary updated

Abstract

This year, we explore generation-augmented retrieval for the TRECVid AVS task. Specifically, the understanding of textual query is enhanced by three generations, including Text2Text, Text2Image, and Image2Text, to address the out-of-vocabulary problem. Using different combinations of them and the rank list retrieved by the original query, we submitted four automatic runs. For manual runs, we use a large language model (LLM) (i.e., GPT4) to rephrase test queries based on the concept bank of the search engine, and we manually check again to ensure all the concepts used in the rephrased queries are in the bank. The result shows that the fusion of the original and generated queries outperforms the original query on TV24 query sets. The generated queries retrieve different rank lists from the original query.
BibTeX
@misc{wu2024polysmart,
  author = {Jiaxin Wu and Chong-Wah Ngo and Xiao-Yong Wei and Qing Li},
  title = {{PolySmart and VIREO @ TRECVid 2024 Ad-hoc Video Search}},
  year = {2024},
  eprint = {2412.15494},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.15494},
}
Figure