Anomaly Detection in Aerial Videos with Transformers

Pu Jin, Lichao Mou, Gui-Song Xia, Xiao Xiang ZhuIEEE TGRS 2022 · DOI · OpenReview

TL;DR

This paper introduces Drone-Anomaly, a new dataset for anomaly detection in aerial videos, and a novel Transformer-based model called ANDT. ANDT leverages self-attention to capture long-range temporal dependencies in video sequences, outperforming existing methods on this challenging task.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly used for critical tasks like inspection, search, and rescue, generating vast amounts of aerial video data. A significant challenge in analyzing this data is identifying rare anomalous events within overwhelmingly normal footage. Manually sifting through hours of video is impractical. Existing anomaly detection methods often struggle with the unique characteristics of aerial videos, such as camera motion and variable resolution, and many are limited in their ability to capture long-term temporal relationships. This paper aims to address these limitations by developing a more robust anomaly detection system for aerial videos and providing a dedicated benchmark dataset.

Methods

The core of the proposed solution is the Anomaly Detection with Transformers (ANDT) model. Unlike traditional methods that might rely on convolutional neural networks (CNNs) with limited receptive fields, ANDT utilizes a Transformer encoder. The key idea is to treat video frames not as individual images but as a sequence of "tubelets" (small spatiotemporal chunks). These tubelets are then projected into embeddings, augmented with positional information, and fed into a Transformer encoder. The self-attention mechanism within the Transformer allows the model to consider relationships between distant parts of the video sequence, capturing long-range temporal dependencies crucial for understanding normal behavior.

The model operates on a future frame prediction principle. During training, ANDT learns to predict the next frame in a sequence based on preceding normal frames. When presented with test data, if the model can accurately predict the next frame, the sequence is considered normal. However, if the prediction error is high, it indicates an unpredictable temporal dynamic, signaling an anomaly. A 3D convolutional decoder is then used to reconstruct the predicted frame.

Datasets

The paper introduces the Drone-Anomaly dataset, a new benchmark specifically designed for anomaly detection in aerial videos. It comprises:

  • 37 training video sequences and 22 testing video sequences.
  • Videos captured from 7 realistic scenes (e.g., highway, crossroads, railway inspection).
  • A total of 87,488 color frames (640x640 resolution at 30 fps).
  • Anomalies are real-world and not staged.

The authors also use the AU-AIR-Anomaly dataset for comparative evaluation.

Key Insights and Ideas

  • Transformers for Temporal Reasoning: The paper highlights the power of Transformer architectures, originally developed for NLP, in capturing long-range temporal dependencies in video data. This is a significant departure from CNN-centric approaches that often struggle with extended temporal contexts.
  • Future Frame Prediction as a Normality Model: Framing anomaly detection as predicting future frames based on learned normality is an effective unsupervised approach. Anomalies are implicitly defined as deviations from predictable temporal patterns.
  • Tubelets for Spatiotemporal Representation: Representing video as a sequence of tubelets allows the Transformer to process both spatial and temporal information simultaneously, preserving crucial spatiotemporal context.
  • The Importance of a Dedicated Aerial Video Dataset: The Drone-Anomaly dataset addresses a critical gap, providing realistic, diverse aerial video scenarios with genuine anomalies, enabling more robust benchmarking.

Key Results

  • ANDT Performance: The proposed ANDT model achieved state-of-the-art or highly competitive results across various scenes in the Drone-Anomaly dataset, often outperforming memory-based and GAN-based methods. For instance, it achieved an AUC of 68.7% on the challenging highway scene and 65.2% on the crossroads scene.
  • AU-AIR-Anomaly Dataset: ANDT also performed exceptionally well on the AU-AIR-Anomaly dataset, achieving an AUC of 86.7%, demonstrating good generalization ability.
  • Ablation Studies: The paper's ablation studies revealed that a patch size of 16x16, 2 Transformer layers, and 6 attention heads yielded optimal performance for ANDT. Using 6 input frames for prediction was found to be the sweet spot, with fewer frames being insufficient and more frames leading to degraded performance.
  • Low False Positive Rate: In a scene with no anomalies, ANDT demonstrated a very low false positive rate (0.0041), indicating its reliability in normal operational scenarios.

Limitations and Future Work

  • Data Scarcity in Specific Scenes: The authors note that in scenes with very limited training data (e.g., railway inspection), performance can be suboptimal, and no single method dominates.
  • Insufficient Training Data Impact: For scenes like vehicle roundabout, the model's performance was affected by insufficient training data, leading to relatively poorer results compared to memory-based methods.
  • Future Work: The authors plan to focus on spatiotemporally detecting anomalous events in aerial videos, suggesting a move towards more fine-grained localization and characterization of anomalies.

Summary updated

Abstract

Unmanned aerial vehicles (UAVs) are widely applied for purposes of inspection, search, and rescue operations by the virtue of low-cost, large-coverage, real-time, and high-resolution data acquisition capacities. Massive volumes of aerial videos are produced in these processes, in which normal events often account for an overwhelming proportion. It is extremely difficult to localize and extract abnormal events containing potentially valuable information from long video streams manually. Therefore, we are dedicated to developing anomaly detection methods to solve this issue. In this paper, we create a new dataset, named DroneAnomaly, for anomaly detection in aerial videos. This dataset provides 37 training video sequences and 22 testing video sequences from 7 different realistic scenes with various anomalous events. There are 87,488 color video frames (51,635 for training and 35,853 for testing) with the size of $640 \times 640$ at 30 frames per second. Based on this dataset, we evaluate existing methods and offer a benchmark for this task. Furthermore, we present a new baseline model, ANomaly Detection with Transformers (ANDT), which treats consecutive video frames as a sequence of tubelets, utilizes a Transformer encoder to learn feature representations from the sequence, and leverages a decoder to predict the next frame. Our network models normality in the training phase and identifies an event with unpredictable temporal dynamics as an anomaly in the test phase. Moreover, To comprehensively evaluate the performance of our proposed method, we use not only our Drone-Anomaly dataset but also another dataset. We will make our dataset and code publicly available. A demo video is available at https://youtu.be/ancczYryOBY. We make our dataset and code publicly available .
BibTeX
@article{jin2022anomaly,
  author = {Pu Jin and Lichao Mou and Gui-Song Xia and Xiao Xiang Zhu},
  title = {{Anomaly Detection in Aerial Videos with Transformers}},
  journal = {IEEE Trans. Geosci. Remote. Sens.},
  year = {2022},
  doi = {10.1109/tgrs.2022.3198130},
}
Figure