Generalization of feature embeddings transferred from different video anomaly detection domains

Fernando Pereira dos Santos, Leonardo Sampaio Ferraz Ribeiro, Moacir A. PontiJVCIR 2019 · DOI · OpenReview

TL;DR

This paper tackles the challenge of adapting video anomaly detection models to new, unseen environments by proposing novel metrics to evaluate how well feature embeddings generalize across different video domains. They demonstrate that Transfer Component Analysis (TCA) significantly improves generalization compared to using raw CNN features or PCA, especially in natural scenarios.

What Problem Does This Paper Address?

Detecting anomalies in video surveillance is crucial, but training effective detectors often requires large amounts of labeled data, which is scarce for specific target domains. Transfer learning offers a solution by leveraging knowledge from source domains. However, a key challenge is understanding how well features learned in one domain will generalize to another, especially when the domains have different characteristics (e.g., lighting, camera angles, types of anomalies). This paper specifically investigates the generalization of pre-trained Convolutional Neural Network (CNN) features for video anomaly detection and proposes methods to quantify this generalization.

Methods

The core of this work lies in proposing and evaluating new metrics for cross-domain generalization. The authors introduce:

  • Feature Extraction: They use pre-trained VGG-19 CNNs to extract feature embeddings from video frames. The idea is that these off-the-shelf features can be transferred without requiring additional labels for the target task.
  • Transfer Learning Techniques:
    • Baseline: Using raw VGG-19 features directly.
  • PCA (Principal Component Analysis): Applying PCA for dimensionality reduction on the feature space.
  • TCA (Transfer Component Analysis): A more advanced technique that aims to find a common subspace where data from different domains are closer, thus reducing the domain gap. This is applied for the first time to this specific anomaly detection problem.
  • Generalization Metrics:
    • Partial Cross-domain Feature Space Generalization (Gpart): Measures the one-way applicability of a model trained on domain A to domain B.
  • Complete Cross-domain Feature Space Generalization (Gcomp): A more comprehensive metric that considers generalization in both directions (A to B and B to A) and provides a more robust assessment of the transfer learning method.

Datasets

The authors evaluate their methods on seven diverse video anomaly detection datasets, categorized into:

  • Natural Scenarios (Water Surveillance):
    • Canoe
    • Boat-River
    • Boat-Sea
  • Urban Scenarios (Traffic and Pedestrian Activities):
    • UCSD Pedestrian 1 (Ped1)
    • UCSD Pedestrian 2 (Ped2)
    • Belleview
    • Train

These datasets vary in frame resolution, number of training frames, illumination, perspective, and clutter, providing a good testbed for cross-domain generalization.

Key Insights and Ideas

  • Domain Gap is Critical: The paper strongly emphasizes that the "domain gap" (differences between source and target domains) is a major hurdle for effective transfer learning in video anomaly detection.
  • CNN Features Generalize, But Not Perfectly: While pre-trained CNN features offer a good starting point, they don't automatically guarantee good performance on vastly different domains.
  • TCA Outperforms PCA and Raw Features: Transfer Component Analysis (TCA) consistently shows superior performance in generalizing features across domains, especially in natural scenarios, compared to using raw VGG-19 features or PCA.
  • Generalization Metrics are Essential: Traditional metrics like AUC and EER are insufficient for evaluating cross-domain transfer. The proposed Gpart and Gcomp metrics provide a more nuanced understanding of how well a model transfers.
  • Anomaly Definition Matters: The paper highlights that the meaning of an anomaly can differ significantly across domains, impacting transferability. Domains with similar anomaly definitions (e.g., Ped1 and Ped2) tend to benefit more from transfer learning.

Key Results

  • Natural Scenarios: TCA achieved an average AUC of 86.68%, significantly outperforming Full VGG-19 (70.47%) and PCA (69.52%). For EER, TCA achieved 16.86% compared to 28.37% for Full VGG-19 and 33.3% for PCA.
  • Urban Scenarios: The performance difference between Full VGG-19 and TCA was less pronounced, with average AUCs of 63.84% and 62.8% respectively. However, PCA showed positive results in urban scenarios, suggesting its utility in certain contexts.
  • Gpart Analysis: TCA generally showed superior Gpart scores, indicating better transferability. For instance, in natural scenarios, TCA's average Gpart AUC was 8.47%, compared to 22.43% for Full VGG-19 and 17.27% for PCA.
  • Gcomp Analysis: TCA also demonstrated better Gcomp scores, confirming its ability to achieve more robust generalization across domains.
  • Negative Transfer: In urban scenarios, particularly when transferring from the "Train" dataset to others (Ped1, Ped2, Belleview), negative transfer was observed, meaning the transfer learning method performed worse than not transferring at all.

Limitations and Future Work

  • Computational Cost of TCA: While TCA shows excellent generalization, its computational time performance can be severely degraded when dealing with a large number of video frames, as seen in the UCSD datasets.
  • GPU Acceleration: The authors suggest that running TCA on GPUs or using multi-core CPUs could significantly improve its performance.
  • Future Directions: The paper suggests exploring these generalization measures for selecting optimal feature extraction methods and for merging diverse datasets to enhance learning guarantees. Further investigation into transfer learning for video activity and other computer vision tasks is also recommended.

Summary updated

Abstract

Highlights • Cross-domain feature space generalization measure quantifies transfer learning. • Simple approach that allows a guideline for feature extraction tools. • Transfer component analysis offers high generalization for video anomaly detection. Abstract Detecting anomalous activity in video surveillance often suffers from limited availability of training data. Transfer learning may close this gap, allowing to use existing annotated data from some source domain. However, analyzing the source feature space in terms of its potential for transfer of learning to another context is still to be investigated. This paper reports a study on video anomaly detection, focusing on the analysis of feature embeddings of pre-trained CNNs with the use of novel cross-domain generalization measures that allow to study how source features generalize for different target video domains. This generalization analysis represents not only a theoretical approach, can be useful in practice as a path to understand which datasets allow better transfer of knowledge. Our results confirm this, achieving better anomaly detectors for video frames and allowing analysis of transfer learning’s positive and negative aspects.
BibTeX
@article{santos2019generalization,
  author = {Fernando Pereira dos Santos and Leonardo Sampaio Ferraz Ribeiro and Moacir A. Ponti},
  title = {{Generalization of feature embeddings transferred from different video anomaly detection domains}},
  journal = {JVCIR},
  year = {2019},
  doi = {10.1016/j.jvcir.2019.02.035},
}
Figure