A Fast Partial Video Copy Detection Using KNN and Global Feature Database

Weijun Tan, Hongwei Guo, Rushuai LiuWACV 2022

TL;DR

This paper introduces a novel framework for fast partial video copy detection (PVCD) by treating it as a video retrieval problem. Instead of brute-force scanning, it leverages a global KNN-searchable database of CNN features from reference videos and enhances frame features with a transformer encoder, achieving state-of-the-art F1 scores and significant speed improvements.

What Problem Does This Paper Address?

The proliferation of online video content has led to a surge in copyright infringement and privacy concerns, making efficient identification of copied video segments crucial. Partial Video Copy Detection (PVCD) aims to find these segments within large databases of untrimmed videos, a task complicated by various video editing techniques like scaling, cropping, and frame rate changes. Traditional PVCD methods often involve scanning reference videos one by one, which is computationally expensive and slow, especially when dealing with deep CNN features that are resource-intensive. This paper addresses the need for a faster and more accurate PVCD system that can handle the challenges of partial copies and leverage the power of deep learning features.

Methods

The proposed PVCD framework consists of three main components:

  • Global Feature Database and Fast KNN Search: The core innovation is organizing all frame CNN features from reference videos into a KNN-searchable global database. When a query video segment is processed, its frame features are used to perform a fast KNN search against this database. This quickly identifies potential candidate videos without scanning each one individually. The scores from the KNN search are aggregated to rank candidate videos.
  • Transformer Encoder for Feature Enhancement: To improve the representation of individual frame features, the authors incorporate a transformer encoder. Inspired by its success in natural language processing and other computer vision tasks, the transformer aggregates temporal context, enhancing the frame features and making them more robust to variations. This is applied to Resnet-29 RMAC features.
  • Modified Temporal Network Localization: For the final step of localizing the exact copy segment within candidate videos, the paper adapts a temporal network algorithm. Key modifications include using only the top-K matched reference frames for each query frame and introducing a "max_diff" constraint to encourage the network flow to follow a diagonal path in the similarity matrix. These changes significantly speed up the localization process while maintaining or improving accuracy.

Datasets

  • VCDB (Video Copy Detection) Dataset: This is the primary dataset used for evaluation. It contains 528 videos in the core dataset with 9236 annotated copy segments. The authors use the core dataset videos as reference videos and the annotated segments as queries.
  • Private PVCD Dataset: A private dataset with over 300 reference videos (longer than VCDB) and thousands of query videos is used for training the transformer encoder. This is necessary because the VCDB dataset is used for benchmark performance testing and cannot be split for training and testing.

Key Insights and Ideas

  • PVCD as a Retrieval Problem: Reframing PVCD as a video retrieval problem, solvable with efficient KNN search on a global feature database, is a significant departure from previous frame-by-frame scanning approaches.
  • Transformer for Temporal Context: Applying transformer encoders to enhance frame-level CNN features proves effective in capturing temporal dependencies, leading to improved similarity representations and better PVCD performance.
  • Optimized Temporal Network: Modifying the temporal network localization by restricting the search space (top-K frames) and enforcing diagonal flow significantly boosts speed without sacrificing accuracy.
  • Reconstructed Similarity Matrix: Using a reconstructed similarity matrix based on KNN results, rather than the full pairwise matrix, leads to comparable or better performance with a substantial speedup in localization.

Key Results

  • State-of-the-Art Performance: The proposed framework achieves a best F1 score of 0.8764 on the VCDB dataset, significantly outperforming previous state-of-the-art methods (e.g., 0.8025).
  • Speed Improvements:
    • The KNN search with FAISS (Flat-CPU, Flat-GPU, IVF256-Flat-GPU) offers speedups of 8x to 15x over brute-force matrix multiplication for query search time.
  • The modified temporal network localization is considerably faster than the original, with one configuration being over 50 times faster.
  • The transformer encoder, while adding computation, allows for smaller parameters in the temporal network, potentially leading to overall speed gains.
  • Feature Importance: The Resnet-29 RMAC feature significantly outperforms VGG-16 features, and the transformer encoder further boosts performance on these strong features.

Limitations and Future Work

  • Transformer Training Data: The authors acknowledge the need for a separate dataset for training the transformer, as the VCDB dataset is reserved for benchmarking.
  • Failed Case: A qualitative example shows a failed detection case where the target video has a relatively small actual content window, indicating that further improvements are needed for such challenging scenarios.
  • Future Work: The paper suggests that the combination of the transformer and temporal network localization brings extra benefits, implying that further exploration of these synergistic effects could lead to even better results. The authors also mention that the transformer can be applied to other CNN features, suggesting potential for broader applicability.

Summary updated

BibTeX
@inproceedings{tan2022fast,
  author = {Weijun Tan and Hongwei Guo and Rushuai Liu},
  title = {{A Fast Partial Video Copy Detection Using KNN and Global Feature Database}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2022},
}
Figure