Introduction
Ad-hoc Video Search (AVS) represents a significant challenge in the field of computer vision and information retrieval: the ability to locate specific video segments using arbitrary, natural language queries. While modern embedding models have made strides, the PolySmart and VIREO team identifies three persistent bottlenecks: out-of-vocabulary (OOV) terms that fall outside a model's training set, complex logical constraints (e.g., "and/or" operations), and nuanced spatial relationships (e.g., "inside" vs. "in front of").
To address these, the team developed a generation-augmented retrieval framework for the TRECVid 2024 AVS task. By leveraging generative AI to "re-imagine" and expand user queries across multiple modalities, they successfully bridged the gap between human intent and machine-searchable video content.
Methodology: Generation-Augmented Retrieval
The core of the proposed system is a plug-in module that augments the original user query through three distinct generative pathways, as illustrated in the system architecture below:
Multi-Modal Query Expansion
- Text-to-Text (T2T): Using Llama 3, the system rephrases queries to align with the search engine's internal "concept bank." This effectively translates OOV words into synonyms that the retrieval model can recognize.
- Text-to-Image (T2I): The query is processed by Stable Diffusion to generate synthetic images. This step is crucial for capturing spatial and logical constraints that are often lost in text-only embeddings.
- Image-to-Text (I2T): These synthetic images are then captioned using BLIP-2, providing a rich, descriptive context that offers a different linguistic perspective on the original query.
Finally, the system retrieves video rankings for all four versions (original + three augmented) and fuses them using a linear combination to produce the final result.
Visualizing Logic and Spatial Constraints
One of the most innovative aspects of this approach is using Text-to-Image generation to resolve complex spatial queries. While a text model might struggle to distinguish between "in front of" and "inside," a generative image model can visually represent these relationships, providing a concrete anchor for the search engine.
As shown in the figure above, the T2I model successfully visualizes logical "or" constraints (e.g., red or blue scarves) and spatial positioning, which helps the retrieval engine focus on the correct visual features.
Results and Performance
The team evaluated their approach on the V3C2 dataset using 30 complex queries. The results, measured in mean xinfAP (Extended Inferred Average Precision), demonstrate the clear advantage of the fusion approach.
| Run ID | Query Type | xinfAP |
|---|---|---|
| Run 1 | Original + T2T + T2I + I2T (Fusion) | 0.294 |
| Run 2 | T2T query only | 0.283 |
| Run 3 | I2T query only | 0.277 |
| Run 4 | Original query only | 0.277 |
| Run Novelty | T2I query only | 0.216 |
The fusion model (Run 1) achieved the highest performance, outperforming the original query baseline by a significant margin. Interestingly, while T2I alone had the lowest individual score, its contribution to the final ensemble was vital for resolving the spatial and logical nuances that text-only models often miss.
Conclusion and Future Directions
The PolySmart and VIREO team's success at TRECVid 2024 highlights the potential of generative AI as a pre-processing step for complex retrieval tasks. By treating LLMs and diffusion models as "translators" and "visualizers," the system overcomes traditional OOV and spatial reasoning hurdles. Future work will focus on improving "query attention" in image-to-video retrieval to ensure specific attributes, such as color and quantity, are preserved more accurately throughout the pipeline.
Summary updated