TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domains

George Awad, Asad A. Butt, Keith Curtis, Jonathan Fiscus, Afzal Godil, Yooyoung Lee, Andrew Delgado, Jesse Zhang, Eliot Godard, Baptiste Chocot, Lukas Diduch, Jeffrey Liu, Alan F. Smeaton, Yvette Graham, Gareth J. F. Jones, Wessel Kraaij, Georges QuenotarXiv 2021 · arXiv

TL;DR

The paper provides a comprehensive overview of the TRECVID 2020 campaign, a premier global benchmarking initiative for video analysis and retrieval. This iteration evaluates 29 international teams across six diverse tasks, introducing two new tracks: Disaster Scene Description and Indexing (DSDI) and Video Summarization (VSUM). The findings highlight a clear shift toward joint visual-textual embedding spaces and underscore the persistent challenges of zero-shot learning and complex action recognition in video.

What Problem Does This Paper Address?

Video data is massive, unstructured, and growing exponentially. Content-based video retrieval—finding specific events, objects, actions, or summaries within thousands of hours of footage—is incredibly difficult. The field historically lacks uniform benchmarks to reliably compare different computer vision and multimodal algorithms.

TRECVID addresses this by providing structured tasks, massive datasets, and standardized human and automated evaluation metrics. In 2020, the campaign expanded beyond traditional search to tackle real-world application domains like automated video captioning, surveillance activity detection, character-centric video summarization, and public safety/disaster response analysis.

Methods

TRECVID does not prescribe a single method; instead, it provides a rigorous evaluation framework for diverse participant methodologies. Key system approaches across the 2020 tasks include:

  • Ad-hoc Video Search (AVS): A clear transition is underway from traditional "concept-based banks" (detecting discrete objects/scenes and combining them queries) to joint visual-textual embedding spaces (e.g., VSE++, dual encoding networks with BERT) to directly match natural language queries to video frames.
  • Instance Search (INS): Systems tackled a complex "person + action" paradigm by combining face recognition (e.g., ArcFace, dlib) with object tracking (SiamRPN) and spatio-temporal action recognition models (SlowFast, TSM, HRNet for pose estimation).
  • Video to Text (VTT): Teams utilized sequence-to-sequence architectures (Transformers, LSTMs) optimized with reinforcement learning loss functions (e.g., Self-Critical Sequence Training) to generate natural language descriptions.

For evaluation, the campaign employs sophisticated metrics like extended inferred average precision (xinfAP) to handle large-scale pooling with sparse judging resources, alongside human Direct Assessment (DA) via Amazon Mechanical Turk for caption quality.

Datasets

  • Vimeo Creative Commons Collection (V3C1 & V3C2): ~1,000 hours each of diverse internet videos segmented into millions of short shots, used for AVS and VTT testing.
  • BBC EastEnders Dataset: 464 hours of a long-running soap opera, providing a "small world" with recurring characters and environments, used for Instance Search and the new Video Summarization task.
  • VIRAT Dataset: ~10-29 hours of high-resolution, realistic outdoor surveillance footage used for the Activities in Extended Video (ActEV) task.
  • Low Altitude Disaster Imagery (LADI): Over 20,000 annotated drone/aircraft images from natural disasters used as training data for the new DSDI task, paired with a new 5-hour pilot test video dataset from the 2015 Nepal earthquake.

Key Insights and Ideas

  • Embeddings Over Concept Banks: In ad-hoc search, joint visual-textual embedding spaces have officially taken the lead over traditional concept-bank approaches due to superior performance, though concept banks remain useful as a complementary ensemble component.
  • Zero-Shot Web Training is Hard: "Type E" runs—systems trained purely on automatically collected web data using only the text query—performed significantly worse than systems utilizing curated training datasets. Zero-shot video retrieval remains an open frontier.
  • Actions Complicate Instance Search: Adding an action requirement (e.g., "Find Max drinking") drastically drops performance compared to previous years' tasks that only required finding a specific person in a specific location. Action recognition remains a major bottleneck.
  • Class Imbalance in Disaster Response: For the new DSDI task, models excelled at identifying common environmental features (grass, trees, buildings) but struggled heavily with rare, critical disaster features (rubble, road washouts) due to severe dataset class imbalance.

Key Results

  • AVS: The best fully automatic runs achieved a mean xinfAP of ~0.35, outperforming manually-assisted runs, which topped out around 0.25.
Figure 2: AVS: 26 Automatic Runs across 20 Main Queries
  • ActEV: The top-performing system (INF-CMU) achieved a normalized partial Area Under the DET Curve (nAUDC) of 0.423, significantly leading the pack (the next best was BUPT-MCPRL at 0.555). "Person rides bicycle" proved to be the easiest activity to detect, while "person crouches" was the hardest.
  • VTT: RUC_AIM3 dominated the video captioning task across all metrics (CIDEr, SPICE, METEOR, BLEU). Human evaluation confirmed that while top systems are improving, none have reached human-level description quality yet.
  • VSUM: In the inaugural video summarization track, the fan-wiki-driven approach by team MeMAD consistently outperformed the purely deep-learning-based framework of NII_UIT by successfully leveraging external textual synopses to identify major life events.

Limitations and Future Work

  • Computational Inefficiency: The authors noted that most top-performing retrieval systems remain computationally heavy and inefficient, highlighting a need for fast, real-time retrieval algorithms.
  • DSDI Dataset Noise: The LADI training dataset suffers from noisy crowd-sourced annotations and extreme class imbalances. Future iterations will focus on cleaner, spatio-temporal annotations and incorporating GIS features based on image location.
  • VTT Evolution: The Matching and Ranking subtask in VTT has reached the end of its usefulness and will be replaced by a new "Fill in the Blank" visual question-answering task to better test contextual video understanding.

Summary updated

Abstract

The TREC Video Retrieval Evaluation (TRECVID) is a TREC-style video analysis and retrieval evaluation with the goal of promoting progress in research and development of content-based exploitation and retrieval of information from digital video via open, metrics-based evaluation. Over the last twenty years this effort has yielded a better understanding of how systems can effectively accomplish such processing and how one can reliably benchmark their performance. TRECVID has been funded by NIST (National Institute of Standards and Technology) and other US government agencies. In addition, many organizations and individuals worldwide contribute significant time and effort. TRECVID 2020 represented a continuation of four tasks and the addition of two new tasks. In total, 29 teams from various research organizations worldwide completed one or more of the following six tasks: 1. Ad-hoc Video Search (AVS), 2. Instance Search (INS), 3. Disaster Scene Description and Indexing (DSDI), 4. Video to Text Description (VTT), 5. Activities in Extended Video (ActEV), 6. Video Summarization (VSUM). This paper is an introduction to the evaluation framework, tasks, data, and measures used in the evaluation campaign.
BibTeX
@misc{awad2021trecvid,
  author = {George Awad and Asad A. Butt and Keith Curtis and Jonathan Fiscus and Afzal Godil and Yooyoung Lee and Andrew Delgado and Jesse Zhang and Eliot Godard and Baptiste Chocot and Lukas Diduch and Jeffrey Liu and Alan F. Smeaton and Yvette Graham and Gareth J. F. Jones and Wessel Kraaij and Georges Quenot},
  title = {{TRECVID 2020: A comprehensive campaign for evaluating video retrieval tasks across multiple application domains}},
  year = {2021},
  eprint = {2104.13473},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.13473},
}
Figure