Abstract
The Ad-hoc Video Search (AVS) task at TRECVID 2019 requires systems to retrieve video segments based on complex natural language queries. This paper describes the joint submission by researchers from Kindai and Kobe Universities, which utilizes a dual-embedding approach to bridge the semantic gap between visual content and textual descriptions. By integrating a coarse-grained global matching model (VSE++) with a fine-grained region-based attention model (SCAN), the system achieves a Mean Average Precision (MAP) of 0.087. Our findings highlight the critical importance of local, fine-grained visual features in satisfying specific search constraints over traditional concept-based detection methods.
Introduction
Ad-hoc Video Search (AVS) remains a cornerstone challenge in computer vision and information retrieval. The goal is to find video shots that match a natural language query, such as "Find shots of a man and a baby both visible." Traditional approaches often relied on "concept-based" detection, where a fixed set of detectors (e.g., "man," "baby," "outdoors") were fused using heuristic rules. However, these methods are limited by the predefined vocabulary of detectors and the complexity of manual fusion logic.
To address these limitations, we propose an embedding-based framework. Instead of explicit concept detection, our system learns to project both video frames and text queries into a shared mathematical space. In this space, the relevance of a video shot to a query is determined by the proximity of their respective vector representations, allowing for direct similarity calculation.
Methodology
Our system architecture combines two distinct embedding strategies to capture both global context and local details.
Coarse-grained Embedding (VSE++)
The coarse-grained model, based on Visual Semantic Embedding (VSE++), focuses on the overall scene. It employs a ResNet152 backbone to extract a 2048-dimensional feature vector from the entire video frame. Simultaneously, a Gated Recurrent Unit (GRU) processes the query sentence into a 1024-dimensional vector. The model is trained using a triplet loss function with hard negative mining, which encourages the model to distinguish between the correct image-caption pair and the most confusingly similar incorrect pairs.
Fine-grained Embedding (SCAN)
The Stacked Cross Attention Network (SCAN) serves as our fine-grained matching component. Unlike the global approach, SCAN decomposes the image into salient regions to identify specific entities.
As shown in Figure 1, the model uses a Faster R-CNN (pre-trained on Visual Genome) to detect up to 36 distinct regions within a frame. An attention mechanism then calculates the similarity between each detected region and every word in the query. This allows the system to verify the presence of specific entities and their relationships, which is often lost in global pooling. The final relevance score is an ensemble of the VSE++ and SCAN scores. The authors also noted that L2-normalizing regional features within the SCAN model provided a performance advantage in specific configurations.
Experiments and Results
The embedding models were trained on the MS-COCO and Flickr 30k datasets. Evaluation was conducted on the TRECVID 2019 AVS dataset, which involves searching across large-scale video archives.
Performance and Ranking
Our best-performing run achieved a Mean Average Precision (MAP) of 0.087. This result placed our team 5th out of 9 participating groups in the "fully automatic" category.
As illustrated in Figure 3, our system demonstrated competitive performance across the board. Notably, our system achieved the highest MAP among all participants for three specific search topics, indicating that our fine-grained approach is particularly effective for certain types of complex, multi-entity queries.
Ablation Study
Ablation experiments confirmed the superiority of the fine-grained approach. The SCAN model alone achieved a MAP of 0.080, significantly outperforming the VSE++ model's MAP of 0.059. The ensemble of both models provided a further boost to 0.087, suggesting that while local details are paramount, global context still provides valuable refinement for final results.
Discussion and Future Work
While effective, the fine-grained SCAN model is computationally intensive, requiring approximately 13.3 hours to process a single search topic. Future work will focus on developing "fast matching" techniques to prune irrelevant shots before applying the expensive attention mechanism. Additionally, we aim to address the "Out-of-Vocabulary" (OOV) problem by leveraging large-scale web image datasets with diverse captions to expand the system's linguistic coverage. Finally, incorporating the structural syntax of queries could further improve the precision of region-to-word matching.
Summary updated