RUC_AIM3 at TRECVID 2020: Ad-hoc Video Search & Video to Text Description

Yida Zhao, Yuqing Song, Shizhe Chen, Qin JinTRECVID 2020 · OpenReview

Introduction

The RUC_AIM3 team from Renmin University of China achieved first place in both the Ad-hoc Video Search (AVS) and Video to Text Description (VTT) tasks at TRECVID 2020. Their approach addresses the "semantic gap" in video understanding—the difficulty of bridging raw pixel data with complex natural language queries. By fusing global semantic matching with fine-grained hierarchical reasoning, the team developed a robust framework for retrieving and describing video content at scale.

Methodology

The core of the RUC_AIM3 strategy is a dual-branch architecture designed to capture both high-level context and specific details.

Dual-Branch Matching

  • Global Matching Branch: This branch utilizes VSE++ and Dual Encoding models to capture the overall semantic context of a video. To enhance linguistic depth, the team integrated BERT as a text encoder.
  • Fine-grained Matching Branch: This branch employs Hierarchical Graph Reasoning (HGR), which decomposes text queries into a hierarchy of events, actions, and entities. These components are then matched against video features at multiple granularities.

Hubness Mitigation and Generation

In high-dimensional retrieval, certain items (hubs) can appear as the nearest neighbor to many queries, reducing accuracy. The team implemented Inverted Softmax during the VTT ranking task to de-bias these results. For video captioning, they used a two-layer LSTM with an attention mechanism processing both scene-level and object-level features (detected via Faster R-CNN). The generation model was further refined using Reinforcement Learning (Self-Critical Sequence Training) to optimize for the CIDEr metric.

Experimental Setup

The models were trained on a diverse set of datasets, including MSRVTT, TGIF, and VATEX for video captioning, and MSCOCO for image-based generalization. Evaluation was conducted on the V3C vimeo collection (over 1 million clips) for AVS and the TRECVID VTT 2016-2020 datasets. Pre-trained features included ResNeXt-101 (2D), irCSN-152 (3D/temporal), and Faster R-CNN (spatial/object).

Key Results and Discussion

The RUC_AIM3 team dominated the competition leaderboards, demonstrating the effectiveness of their hybrid approach.

Ad-hoc Video Search (AVS)

As shown in the results, the combination of global and fine-grained matching significantly outperformed previous benchmarks. Their best run achieved an infAP of 0.359, more than doubling the 2019 winner's score.

AVS performance comparison across different model configurations and training data.

Video to Text Description (VTT)

In the VTT task, the team secured first place in both matching and generation. For generation, their ensemble model reached a BLEU@4 of 5.56 and a CIDEr of 30.30.

Performance metrics for VTT generation runs on the TRECVID 2020 dataset.

Insights on Model Architecture

  • LSTMs vs. Transformers: Interestingly, LSTMs outperformed Transformers for video captioning. The authors suggest that because video captions are typically short, visual grounding is more critical than the long-range textual modeling where Transformers excel.
  • Temporal Dominance: Models utilizing temporal (scene-level) attention generally outperformed those relying solely on spatial (object-level) attention, indicating that actions are often more descriptive than static objects in a video context.

Conclusion and Future Work

While the dual-branch framework proved highly effective, the team noted that including additional image data (MSCOCO) yielded inconsistent results due to domain shifts between images and video. Future research will focus on refining hybrid reranking strategies, using separate language models and video-text matchers to further improve the fluency and relevance of generated descriptions.

Summary updated

BibTeX
@inproceedings{zhao2020ruc,
  author = {Yida Zhao and Yuqing Song and Shizhe Chen and Qin Jin},
  title = {{RUC\_AIM3 at TRECVID 2020: Ad-hoc Video Search \& Video to Text Description}},
  booktitle = {TRECVID},
  year = {2020},
  url = {https://openreview.net/forum?id=G-jImwVp0iM},
}
Figure