A Causal Inference Look at Unsupervised Video Anomaly Detection

Xiangru Lin, Yuyang Chen, Guanbin Li, Yizhou YuAAAI 2022 · OpenReview

Abstract

Unsupervised video anomaly detection, a task that requires no labeled normal/abnormal training data in any form, is challenging yet of great importance to both industrial applications and academic research. Existing methods typically follow an iterative pseudo label generation process. However, they lack a principled analysis of the impact of such pseudo label generation on training. Furthermore, the long-range temporal dependencies also has been overlooked, which is unreasonable since the definition of an abnormal event depends on the long-range temporal context. To this end, first, we propose a causal graph to analyze the confounding effect of the pseudo label generation process. Then, we introduce a simple yet effective causal inference based framework to disentangle the noisy pseudo label's impact. Finally, we perform counterfactual based model ensemble that blends long-range temporal context with local image context in inference to make final anomaly detection. Extensive experiments on six standard benchmark datasets show that our proposed method significantly outperforms previous state-of-the-art methods, demonstrating our framework's effectiveness.

Introduction

Video Anomaly Detection (VAD) is the task of identifying unusual events, such as accidents or erratic behavior, in video streams. While many systems rely on supervised learning, Unsupervised Video Anomaly Detection (UVAD) is a more practical but difficult frontier where training data is completely unlabeled and contains both normal and abnormal events.

Existing UVAD methods typically rely on iterative pseudo-labeling: guessing which frames are abnormal and training on those guesses. This approach suffers from two major flaws:

  1. The Confounding Trap: Noisy pseudo-labels create "spurious correlations," leading the model to misidentify normal features as abnormal.
  2. Short-Sightedness: Most models only consider local temporal windows, failing to recognize anomalies that are only apparent against a long-range context (e.g., a person running in a quiet library over a 10-minute period).

Methodology

To overcome these issues, the authors propose a two-stage causal inference pipeline.

De-confounded Training (DCFD)

The authors utilize a Causal Graph to identify the noisy pseudo-labeling process as a "confounder"—a variable that creates misleading correlations between input features and predictions. To mitigate this, they implement Backdoor Adjustment. By stratifying the feature space into confounder sets, the model is forced to learn the direct causal effect of the video frame on the anomaly score, effectively blocking the negative influence of incorrect pseudo-labels.

The proposed causal framework: (a) illustrates the causal graph with intervention on the confounder S, and (b) shows the de-confounded training pipeline including confounder set and pseudo-label generation.

Counterfactual Temporal Context Ensemble (CTCE)

To address the lack of long-range context, the framework uses Counterfactual Reasoning during inference. For any given frame, the model asks: "What would the anomaly score be if the local feature were replaced by the average feature of a long-range window (e.g., 1024 frames)?"

Visualization of feature distributions across different temporal window sizes (d). As the window size increases, the features become more stable, providing a robust counterfactual baseline for detecting local anomalies.

By comparing the actual frame to this stable long-range "counterfactual" baseline, the model can better distinguish true anomalies from local noise. This ensemble happens entirely during inference and requires no additional training.

Experimental Results

The model was evaluated on six standard VAD benchmarks, including UCSD Ped1/Ped2, Subway, UMN, and Avenue. The proposed method significantly outperformed previous state-of-the-art (SOTA) unsupervised methods and even rivaled supervised approaches:

  • UCSD Ped2: Achieved 99.4% AUC, a substantial improvement over the previous unsupervised SOTA.
  • Avenue: Reached 90.3% AUC, surpassing the previous best of 85.3%.
  • Architectural Robustness: The framework showed consistent gains regardless of the backbone used (ResNet-50, I3D, or C3D), proving that the causal logic is the primary driver of performance.

Conclusion

This research demonstrates that UVAD is not just a pattern recognition problem but a causal one. By mathematically adjusting for confounding bias and leveraging long-range counterfactual context, the proposed framework sets a new benchmark for unsupervised detection. While the model handles noise well, future work could explore more complex causal graphs, such as those incorporating object-level interactions, to further refine performance in dense environments.

Summary updated

Abstract

Unsupervised video anomaly detection, a task that requires no labeled normal/abnormal training data in any form, is challenging yet of great importance to both industrial applications and academic research. Existing methods typically follow an iterative pseudo label generation process. However, they lack a principled analysis of the impact of such pseudo label generation on training. Furthermore, the long-range temporal dependencies also has been overlooked, which is unreasonable since the definition of an abnormal event depends on the long-range temporal context. To this end, first, we propose a causal graph to analyze the confounding effect of the pseudo label generation process. Then, we introduce a simple yet effective causal inference based framework to disentangle the noisy pseudo label's impact. Finally, we perform counterfactual based model ensemble that blends long-range temporal context with local image context in inference to make final anomaly detection. Extensive experiments on six standard benchmark datasets show that our proposed method significantly outperforms previous state-of-the-art methods, demonstrating our framework's effectiveness.
BibTeX
@inproceedings{lin2022causal,
  author = {Xiangru Lin and Yuyang Chen and Guanbin Li and Yizhou Yu},
  title = {{A Causal Inference Look at Unsupervised Video Anomaly Detection}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  url = {https://openreview.net/forum?id=3H27nHptcDq},
}
Figure