FIVR: Fine-grained Incident Video Retrieval

Giorgos Kordopatis-Zilos, Symeon Papadopoulos, Ioannis Patras, Ioannis Kompatsiaris

Introduction

Video retrieval research has traditionally focused on two distinct ends of the spectrum: Near-Duplicate Video Retrieval (NDVR), which identifies nearly identical copies of a video, and Event Retrieval, which categorizes videos into broad semantic classes like "weddings" or "protests." However, a critical gap exists in the "middle ground": the ability to find different videos of the exact same incident, such as a specific plane crash or a localized terror attack.

This paper introduces Fine-grained Incident Video Retrieval (FIVR), a task designed to bridge this gap. Unlike broad event retrieval, FIVR requires finding all videos related to a specific spatio-temporal span across different viewpoints and time segments. To facilitate this research, the authors present FIVR-200K, a large-scale dataset comprising 225,960 videos, and provide a comprehensive benchmark of existing state-of-the-art methods.

Formalizing the Incident Retrieval Task

The authors argue that an "incident" is defined by a specific spatio-temporal span, allowing for a more rigorous mathematical definition than general semantic similarity. To evaluate retrieval performance effectively, they define three distinct levels of video association:

  1. Duplicate Scene Videos (DSV): These are videos sharing at least one scene captured by the same camera. This category includes near-duplicates but also extends to videos that share only a small segment of footage.
  2. Complementary Scene Videos (CSV): These videos depict the same segment of an incident but are captured from different cameras or viewpoints (e.g., two different bystanders filming the same explosion).
  3. Incident Scene Videos (ISV): These videos belong to the same incident but share no temporal overlap. For example, one video might show the start of a fire, while another shows the aftermath or the rescue operations hours later.

The FIVR-200K Dataset

To support the FIVR task, the authors curated FIVR-200K, which contains 225,960 videos associated with 4,687 Wikipedia news events from 2013 to 2017. The collection process involved crawling Wikipedia’s "Current Events" pages, filtering for disasters and conflicts, and using the headlines as YouTube search queries.

To ensure the dataset's quality and difficulty, the authors implemented a semi-automatic annotation process. They selected 100 challenging queries and manually annotated thousands of results based on the DSV, CSV, and ISV definitions. A key innovation in their collection process was the use of "uploader diversity." By filtering for high uploader ratios, the authors ensured the dataset captured major news events documented by many different individuals, rather than just clusters of duplicates from a single source. This makes FIVR-200K significantly more realistic and challenging than older datasets like CC_WEB_VIDEO, which are smaller and dominated by near-duplicates.

Benchmarking and Methodology

The paper evaluates several feature extraction and aggregation schemes to establish a baseline for the FIVR task:

  • Global Vectors (GV): A simple approach where frame descriptors are averaged into a single vector.
  • Bag-of-Words (BoW): A traditional method using visual word quantization.
  • Layer Bag-of-Words (LBoW): A variant that utilizes intermediate CNN features mapped to a codebook.
  • Deep Metric Learning (DML): Networks trained specifically to minimize the distance between related video representations.
  • Hashing Codes (HC): Projecting frames into Hamming space to enable fast, large-scale retrieval.

Key Results and Insights

The experimental results highlight the significant difficulty of the FIVR task compared to traditional NDVR:

  • Performance Leaders: The LBoW (Layer Bag-of-Words) approach using VGG features emerged as the top performer. It achieved a mean Average Precision (mAP) of 0.710 for the Duplicate Scene Video Retrieval (DSVR) task.
  • The Difficulty Gap: Performance dropped significantly on the more complex tasks, falling to 0.572 for the Incident Scene Video Retrieval (ISVR) task. This confirms that linking videos with zero visual overlap is the most significant challenge in the field.
  • Benchmark Realism: Methods that achieved near-perfect scores (0.95+) on older, simpler datasets saw their performance plummet on FIVR-200K. This underscores the dataset's value as a benchmark for real-world applications where "distractor" videos (similar-looking but unrelated incidents) are common.
  • Deep Features vs. Handcrafted: Deep learning-based features (CNNs) consistently outperformed handcrafted features like SIFT or HSV histograms by substantial margins across all categories.

Conclusion and Future Directions

The introduction of FIVR and the FIVR-200K dataset reveals that current global video representations have reached a performance ceiling. The authors suggest several avenues for future research to address the complexities of incident retrieval:

  • Temporal Alignment: Moving beyond global descriptors toward frame-level matching to better identify short overlapping segments.
  • Query Expansion: Utilizing initially retrieved relevant videos to find additional related content, which may help bridge the gap in ISV tasks where visual overlap is minimal.
  • Multimodal Context: Incorporating audio or text metadata to provide context in cases where visual cues are insufficient to link two videos of the same incident.

By formalizing the "middle ground" of video retrieval, this work provides a vital foundation for journalists, forensic investigators, and human rights researchers who require comprehensive tools to reconstruct complex real-world events.

Summary updated

Abstract

This paper introduces the problem of Fine-grained Incident Video Retrieval (FIVR). Given a query video, the objective is to retrieve all associated videos, considering several types of associations that range from duplicate videos to videos from the same incident. FIVR offers a single framework that contains several retrieval tasks as special cases. To address the benchmarking needs of all such tasks, we construct and present a large-scale annotated video dataset, which we call FIVR-200K, and it comprises 225,960 videos. To create the dataset, we devise a process for the collection of YouTube videos based on major news events from recent years crawled from Wikipedia and deploy a retrieval pipeline for the automatic selection of query videos based on their estimated suitability as benchmarks. We also devise a protocol for the annotation of the dataset with respect to the four types of video associations defined by FIVR. Finally, we report the results of an experimental study on the dataset comparing five state-of-the-art methods developed based on a variety of visual descriptors, highlighting the challenges of the current problem.
BibTeX
@article{kordopatiszilos2018fivr,
  author = {Giorgos Kordopatis-Zilos and Symeon Papadopoulos and Ioannis Patras and Ioannis Kompatsiaris},
  title = {{FIVR: Fine-grained Incident Video Retrieval}},
  journal = {IEEE Trans. Multimedia},
  year = {2018},
  doi = {10.1109/tmm.2019.2905741},
}
Figure