Building Scalable Video Understanding Benchmarks through Sports

Aniket Agarwal, Alex Zhang, Karthik Narasimhan, Igor Gilitschenski, Vishvak Murahari, Yash Kant

TL;DR

This paper introduces ASAP, an automated pipeline for aligning sports videos with web-based commentary, enabling the creation of large-scale, densely annotated video datasets at minimal cost. They leverage this to build LCric, a new benchmark for long-horizon video understanding using over 1000 hours of cricket footage, which reveals significant performance gaps in current state-of-the-art models.

What Problem Does This Paper Address?

Evaluating models on long-horizon video understanding (LVU) is crucial for real-world applications, but existing benchmarks are limited by their scale and annotation quality. Manually annotating long videos (hours-long matches) is prohibitively expensive and time-consuming. While some approaches use subtitles or metadata, they often lack the dense, fine-grained temporal information needed to truly test complex reasoning over extended periods. The authors aim to bridge this gap by creating a scalable method to automatically generate such datasets.

Methods

The core innovation is the Annotation and Video Stream Alignment Pipeline (ASAP). This pipeline works in two main stages:

  1. Match State Extraction: For sports like cricket, ASAP uses OCR to detect and extract the "match state" (e.g., current over and ball) from scorecards embedded in video frames. For other sports like American Football or Basketball, it uses the on-screen timer. This extracted match state acts as a temporal anchor.
  2. Annotation Alignment: Using the temporal anchors, ASAP aligns freely available web annotations (like sports commentary) with the corresponding video segments. This process is designed to be generalizable across different sports.

The authors then use this automated pipeline to create LCric, a large-scale benchmark for LVU, focusing on cricket. LCric consists of over 1000 hours of densely annotated cricket videos, automatically labeled with play-by-play events. They also automatically generate compositional queries (binary, multiple-choice, and regression) to test various LVU capabilities.

Datasets

  • LCric: A newly introduced benchmark for long-horizon video understanding. It comprises 1008 hours of cricket videos, automatically annotated with play-by-play events derived from web commentary. The dataset is segmented into 10-over (~50 minute) clips.
  • Existing Datasets (for comparison): The paper references and compares LCric's scale to existing LVU datasets like VidSitu, VideoStory, MSR-VTT, Charades, TGIF, TVQA, VTW, MovieClips, LVU, YouCook II, and ActNet Captions, highlighting LCric's significantly longer average clip length.

Key Insights and Ideas

  • Automated annotation is key to scaling LVU: The success of ASAP demonstrates that automated pipelines can unlock the creation of massive, densely annotated video datasets, overcoming the bottleneck of manual annotation.
  • Sports commentary is a rich, underutilized source for LVU: By aligning readily available sports commentary with video, the authors unlock a wealth of structured temporal information that can be used for fine-grained event understanding.
  • Current LVU models struggle with long horizons: The performance drop observed in state-of-the-art models on LCric, especially on longer clips, highlights a significant gap between current capabilities and the demands of truly long-horizon reasoning.
  • Compositional queries are essential for robust LVU evaluation: The use of automatically generated queries that require combining information over extended periods provides a more comprehensive evaluation than simple event classification.

Key Results

  • ASAP Accuracy: The ASAP pipeline achieves high alignment accuracy, with human studies showing an average of 95.3% accuracy across four different sports.
  • LCric Scale: The LCric dataset contains over 1008 hours of cricket videos, automatically annotated at virtually zero cost.
  • Model Performance: State-of-the-art LVU models (TQN and MeMViT) show a significant performance degradation on LCric as clip length increases, dropping substantially below a human baseline (around 38% lower on 50-minute clips).
  • Human Baseline: Human annotators achieve high accuracy (over 96%) on LCric queries, demonstrating the dataset's challenging nature for current models.
  • Event Localization Importance: The results suggest that both accurate event localization and robust event detection are crucial for LVU performance.

Limitations and Future Work

  • American Football Alignment Issues: The authors note that alignment accuracy for American Football is slightly lower due to inconsistencies in how timestamps are provided for certain events (e.g., touchdowns vs. play starts).
  • Generalizability of ASAP: While ASAP is shown to work across four sports, its effectiveness might depend on the availability and structure of online commentary and scorecards for other sports.
  • Future Work: The authors hope their work will spur further research into improving LVU models, especially for long-horizon tasks. They also suggest that ASAP can be leveraged to generate large-scale video datasets for other sports, further expanding the scope of LVU research.

Summary updated

Abstract

Existing benchmarks for evaluating long video understanding falls short on two critical aspects, either lacking in scale or quality of annotations. These limitations arise from the difficulty in collecting dense annotations for long videos, which often require manually labeling each frame. In this work, we introduce an automated Annotation and Video Stream Alignment Pipeline (abbreviated ASAP). We demonstrate the generality of ASAP by aligning unlabeled videos of four different sports with corresponding freely available dense web annotations (i.e. commentary). We then leverage ASAP scalability to create LCric, a large-scale long video understanding benchmark, with over 1000 hours of densely annotated long Cricket videos (with an average sample length of ~50 mins) collected at virtually zero annotation cost. We benchmark and analyze state-of-the-art video understanding models on LCric through a large set of compositional multi-choice and regression queries. We establish a human baseline that indicates significant room for new research to explore. Our human studies indicate that ASAP can align videos and annotations with high fidelity, precision, and speed. The dataset along with the code for ASAP and baselines can be accessed here: https://asap-benchmark.github.io/.
BibTeX
@misc{agarwal2023building,
  author = {Aniket Agarwal and Alex Zhang and Karthik Narasimhan and Igor Gilitschenski and Vishvak Murahari and Yash Kant},
  title = {{Building Scalable Video Understanding Benchmarks through Sports}},
  year = {2023},
  eprint = {2301.06866},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2301.06866},
}
Figure