Comprehensive Feature-Based Robust Video Fingerprinting Using Tensor Model

Xiushan Nie, Yilong Yin, Jiande Sun, Ju Liu, Chaoran Cui

TL;DR

This paper introduces a novel video fingerprinting technique that leverages tensor decomposition to create a "comprehensive feature" by fusing multiple video characteristics. This approach significantly improves robustness against combined video modifications, outperforming existing methods by capturing the inherent relationships and consensus between different features.

What Problem Does This Paper Address?

The explosion of digital video content online has led to a proliferation of near-duplicate videos, often altered through various editing techniques. Detecting these near-duplicates is crucial for applications like content-based search, copyright enforcement, and online video monitoring. Existing video fingerprinting methods, which typically rely on a single type of feature (e.g., color histograms) or simple concatenation of features, struggle when videos undergo multiple modifications simultaneously. These combined attacks can degrade the performance of single-feature methods because they fail to capture the complex interplay and agreement between different visual cues. The paper aims to address this limitation by developing a more robust fingerprinting method that can effectively handle combined modifications.

Methods

The core of the proposed method is the creation of a "comprehensive feature" using a tensor model. Here's a breakdown:

  • Video Tensor Construction: Each video is represented as a third-order tensor. This tensor is constructed by stacking different features (e.g., global, local, temporal) extracted from the video. Each feature type forms a "mode" of the tensor.
  • Tensor Decomposition (Tucker Model): The paper employs the Tucker decomposition model to factorize the video tensor. This process decomposes the tensor into a "core tensor" and several "factor matrices," effectively capturing the principal components and intrinsic characteristics of the video content across different features. The Tucker model is chosen for its flexibility and ability to handle interactions between modes.
  • Comprehensive Feature Generation: The comprehensive feature is derived from the factor matrices obtained from the Tucker decomposition. Specifically, the absolute values of the elements in each factor matrix are averaged row-wise to create component vectors. These component vectors are then concatenated to form the final comprehensive feature vector, which serves as the video fingerprint.
  • Matching Strategy: An auxiliary matching strategy is proposed, utilizing the "core tensor" from the decomposition. This strategy aims to narrow down the search space in a large video fingerprint database, thereby accelerating the matching process.

Datasets

The experiments were conducted using videos from the CC_WEB_VIDEO Dataset and the OV Dataset. The authors applied a variety of single-mode and combined modifications to these videos to create a test database of approximately 20,000 videos.

Key Insights and Ideas

  • Beyond Simple Concatenation: The paper highlights that simply concatenating features or using weighted combinations is insufficient. The true power lies in capturing the assistance and consensus among different features, which is what the tensor model excels at.
  • Tensor Model for Feature Fusion: Representing video features as a multi-dimensional tensor allows for a more holistic representation, enabling the capture of complex inter-feature relationships that are missed by traditional methods.
  • Comprehensive Feature as Intrinsic Representation: The comprehensive feature, derived from tensor decomposition, is argued to represent the intrinsic and principal information of the video, effectively filtering out noise and irrelevant details.
  • Robustness to Combined Modifications: The primary insight is that by fusing features in a way that respects their interdependencies (via tensor decomposition), the resulting fingerprints are significantly more robust to combined attacks than those based on single features or simple fusion.

Key Results

  • Superior Robustness: The proposed comprehensive feature-based method demonstrated superior performance across various single-mode modifications compared to existing methods like LRTA, 3D-DCT, and CGO.
  • Significant Improvement under Combined Modifications: The most notable improvement was observed under combined modifications, where the proposed method significantly outperformed the others. For instance, under "combined modification 2" (crop+flip+insertion of patterns+picture in picture+shift), the proposed method achieved an F-score of 0.8292, compared to 0.5745 for 3D-DCT.
  • Faster Matching: The proposed matching strategy, utilizing the core tensor for pre-matching, significantly reduced the matching time compared to an exhaustive search.
  • Statistical Validation: ROC curves and F-score analysis confirmed the robustness and discriminative power of the proposed method.

Limitations and Future Work

  • Feature Selection: The paper acknowledges that the choice of input features is crucial for the performance of the proposed framework. While the method is flexible in accepting various features, the optimal selection of these features is an area for future investigation.
  • Computational Cost: While the matching strategy offers speedups, the tensor decomposition itself can be computationally intensive.
  • Future Work: The authors suggest investigating feature selection strategies to further optimize the system. They also plan to explore other tensor decomposition models and advanced matching techniques.

Summary updated

Abstract

Content-based near-duplicate video detection (NDVD) is essential for effective search and retrieval, and robust video fingerprinting is a good solution for NDVD. Most existing video fingerprinting methods use a single feature or concatenate different features to generate video fingerprints, and show good performance under single-mode modifications such as noise addition and blurring. However, when they suffer combined modifications, the performance is degraded to a certain extent because such features cannot characterize the video content completely. By contrast, the assistance and consensus among different features can improve the performance of video fingerprinting. Therefore, in the present study, we mine the assistance and consensus among different features based on a tensor model, and we present a new comprehensive feature to fully use them in the proposed video fingerprinting framework. We also analyze what the comprehensive feature really is for representing the original video. In this framework, the video is initially set as a high-order tensor that consists of different features, and the video tensor is decomposed via the Tucker model with a solution that determines the number of components. Subsequently, the comprehensive feature is generated by the low-order tensor obtained from tensor decomposition. Finally, the video fingerprint is computed using this feature. A matching strategy used for narrowing the search is also proposed based on the core tensor. The robust video fingerprinting framework is resistant not only to single-mode modifications but also to their combination.
BibTeX
@article{nie2017comprehensive,
  author = {Xiushan Nie and Yilong Yin and Jiande Sun and Ju Liu and Chaoran Cui},
  title = {{Comprehensive Feature-Based Robust Video Fingerprinting Using Tensor Model}},
  journal = {IEEE Trans. Multimedia},
  year = {2017},
  url = {https://openreview.net/forum?id=CkmalIZfIc},
}
Figure