Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection

Jash Dalvi, Ali Dabouei, Gunjan Dhanuka, Min XuWACV 2025

TL;DR

This paper tackles the challenge of weakly-supervised video anomaly detection (VAD) by proposing a novel approach called DAKD. DAKD aggregates knowledge from multiple pre-trained "teacher" models into a single, efficient "student" model, achieving state-of-the-art performance on challenging benchmarks. This work highlights the power of knowledge distillation and multi-backbone aggregation for improving VAD in scenarios with limited labeled data.

What Problem Does This Paper Address?

Video anomaly detection (VAD) aims to automatically identify unusual events in surveillance footage. This task is notoriously difficult due to several factors:

  • Limited Training Data: Datasets are often small, making it hard for models to learn robust representations.
  • Weak Supervision: Typically, only video-level labels (e.g., "anomaly present" or "not present") are available, not precise frame-level annotations of when the anomaly occurs.
  • Class Imbalance: Anomalous events are inherently rare compared to normal activities, creating a significant imbalance in the data.

Existing weakly-supervised VAD methods often struggle with these challenges. While knowledge transfer from pre-trained models has shown promise, effectively combining knowledge from multiple diverse sources and distilling it into an efficient model remains an open problem.

Methods

The proposed DAKD (Distilling Aggregated Knowledge with Disentangled Attention) framework consists of two main stages:

  1. Teacher Model (Aggregated Knowledge):
    • Feature Extraction: Multiple pre-trained video backbones (e.g., I3D, S3D, CLIP) are used to extract diverse features from video clips. The idea is that different backbones capture complementary information.
  2. Temporal Aggregation Module (TAM): A novel module that uses a disentangled attention mechanism to fuse the features from multiple backbones. This TAM specifically incorporates relative positional information between frames and uses both self- and cross-attention to create a comprehensive aggregated representation.
  3. Student Model (Knowledge Distillation):
    • Bi-level Fine-grained Knowledge Distillation: To overcome the computational cost of using multiple backbones, the knowledge from the complex Teacher model is distilled into a single-backbone Student model (using CLIP). This distillation happens at two levels:
  4. Prediction-level Distillation: The Student model's output distribution is aligned with the Teacher's "soft pseudo-labels" (derived from the Teacher's predictions, refined with moving average filtering and min-max normalization). This uses a cross-entropy loss.
  5. Feature-level Distillation: The Student's internal feature representations are aligned with the Teacher's using the InfoNCE loss, which encourages similar representations for corresponding "normal" and "anomaly" features.

Datasets

The authors evaluated their approach on three widely-used benchmark datasets for weakly-supervised VAD:

  • UCF-Crime: Contains 1900 untrimmed videos with video-level labels and 13 types of anomalies.
  • ShanghaiTech: Features 437 videos from fixed-angle street cameras.
  • XD-Violence: A large-scale dataset with 4754 untrimmed videos across diverse scenes.

All datasets provide video-level labels for training and frame-level labels for testing.

Key Insights and Ideas

  • The Power of Aggregation: Combining features from multiple diverse pre-trained backbones significantly boosts performance, suggesting that no single backbone captures all necessary information for VAD.
  • Disentangled Attention for Temporal Context: The proposed TAM effectively leverages disentangled attention to capture both spatial content and temporal relationships, including relative positional information, which is crucial for understanding event sequences.
  • Knowledge Distillation is Key for Efficiency: The bi-level distillation approach successfully transfers the rich knowledge of the multi-backbone Teacher to a computationally efficient single-backbone Student, making the method practical for real-world deployment.
  • Soft Pseudo-Labels Enhance Training: Using the Teacher's refined predictions as soft pseudo-labels for the Student model provides a more robust learning signal than relying solely on hard video-level labels.

Key Results

DAKD achieved state-of-the-art performance on all tested datasets:

  • UCF-Crime: Achieved an AUC of 88.34%, outperforming previous methods by a significant margin (e.g., 1.36% improvement over MGFN).
  • ShanghaiTech: Achieved an AUC of 98.10%, demonstrating strong performance even on datasets where previous methods were nearing saturation.
  • XD-Violence: Achieved an AP score of 85.61%, also outperforming existing methods.

The paper also includes ablation studies showing the effectiveness of individual components, such as the TAM and the different distillation losses.

Limitations and Future Work

  • Computational Cost of Teacher: While the distillation process makes the Student model efficient, training the multi-backbone Teacher model is still computationally intensive.
  • Generalization to New Anomalies: Like most VAD methods, DAKD's performance might be limited when encountering entirely novel anomaly types not represented in the training data.

The authors suggest that their approach provides a strong foundation for future work in weakly-supervised VAD, particularly in exploring more advanced aggregation and distillation techniques. The focus on efficiency and performance in limited-data scenarios is a significant contribution.

Summary updated

BibTeX
@inproceedings{dalvi2025distilling,
  author = {Jash Dalvi and Ali Dabouei and Gunjan Dhanuka and Min Xu},
  title = {{Distilling Aggregated Knowledge for Weakly-Supervised Video Anomaly Detection}},
  booktitle = {IEEE/CVF Winter Conference on Applications of Computer Vision},
  year = {2025},
}
Figure