UTS_CAI submission at TRECVID 2018 Ad-hoc Video Search Task

Fengda Zhu, Xiaohan Wang, Linchao Zhu, Xuanyi Dong, Yi YangTRECVID 2018 · OpenReview

Introduction

Ad-hoc Video Search (AVS) is a core task in the TREC Video Retrieval Evaluation (TRECVID) series, challenging researchers to retrieve relevant video clips from a large-scale dataset based on complex natural language queries. Unlike standard video classification, which relies on a predefined set of labels, AVS requires a "zero-shot" approach where the system must generalize to unseen queries. This task is particularly difficult due to the "semantic gap"—the disconnect between low-level visual features (pixels) and high-level linguistic concepts.

Methodology

The UTS_CAI team developed a dual-encoder framework designed to map both video content and textual queries into a shared, joint semantic embedding space. This allows retrieval to be performed by calculating the distance between vectors in this space.

Video and Language Encoding

For video representation, the system samples frames at a rate of 5 frames per second. Features are extracted using state-of-the-art Convolutional Neural Networks (CNNs), specifically ResNeXt101 and PNAS. These frame-level features are then aggregated via average pooling to form a single representative vector for the entire video clip.

On the language side, queries are first tokenized and converted into word embeddings using Word2Vec. To capture the sequential and syntactic structure of the natural language descriptions, the team explored two types of encoders: Long Short-Term Memory (LSTM) networks and Temporal Convolutional Networks (TCNs).

Visual-Semantic Mapping

The core of the learning process involves a triplet ranking loss function. This objective function encourages the model to minimize the distance between a video and its corresponding ground-truth description while maximizing the distance from "hard negatives"—incorrect descriptions that are semantically or visually similar to the correct ones. Hard negative mining is a critical component of this strategy, as it forces the model to learn fine-grained distinctions.

Ensemble Strategy

To improve robustness and overall performance, the team employed a late fusion ensemble strategy. By averaging the retrieval scores from multiple models trained with different visual extractors and language encoders, the system can leverage the complementary strengths of diverse architectures.

Datasets

The models were trained and validated on several large-scale video-text datasets:

  • MSR-VTT: A large-scale dataset containing approximately 50 hours of video and 260,000 clip-sentence pairs.
  • TGIF (Tumblr GIF): A dataset of 100,000 animated GIFs with 120,000 natural language descriptions.
  • TRECVID 2017/2018: The Internet Archive (IACC.3) dataset was used for validation and the final competition test.

Results and Discussion

The UTS_CAI team submitted four distinct runs to the TRECVID 2018 AVS task to evaluate the impact of different model components and ensemble configurations.

Table 1: Performance comparison across four experimental runs using different model configurations.

As shown in Table 1, the ensemble approaches (RUN 2 and RUN 4) significantly outperformed the single-model baselines. RUN 2, which ensembled two different visual models (ResNeXt101 and PNAS), and RUN 4, which ensembled different language models (LSTM and TCN), both achieved a mean extended inferred average precision (infAP) of 0.067.

The results highlight several key insights:

  1. Architectural Diversity: Combining different types of encoders (e.g., CNN variants or RNN vs. TCN) provides a substantial performance boost.
  2. TCN Effectiveness: RUN 3 demonstrates that TCNs are powerful language encoders for video retrieval, achieving an infAP of 0.058 on their own, which is significantly higher than the basic LSTM baseline in RUN 1 (0.031).
  3. Competitive Performance: While an infAP of 6.7% may appear low, it was a competitive result for the challenging zero-shot AVS task in 2018, reflecting the inherent difficulty of the problem.

Conclusion

The UTS_CAI submission demonstrates the potential of dual-encoder frameworks and triplet ranking loss for ad-hoc video search. The use of diverse ensembles and hard negative mining proved essential for achieving competitive results. Future research will focus on better capturing temporal dynamics within video clips and exploring more sophisticated fusion mechanisms, such as attention-based alignment between visual regions and specific words in the query.

Summary updated

BibTeX
@inproceedings{zhu2018uts,
  author = {Fengda Zhu and Xiaohan Wang and Linchao Zhu and Xuanyi Dong and Yi Yang},
  title = {{UTS\_CAI submission at TRECVID 2018 Ad-hoc Video Search Task}},
  booktitle = {TRECVID},
  year = {2018},
  url = {https://openreview.net/forum?id=LGAhWxVnlj},
}
Figure