Disentangled Dynamic Deviation Transformer Networks for Multivariate Time Series Anomaly Detection

Chunzhi Wang, Shaowen Xing, Rong Gao, Lingyu Yan, Naixue Xiong, Ruoxi Wang2023 · DOI

TL;DR

This paper introduces D³TN, a novel deep learning architecture for detecting anomalies in multivariate time series data. D³TN excels by disentangling and modeling both the dynamic relationships between sensors and long-term temporal dependencies, leading to significantly improved anomaly detection accuracy compared to existing methods.

What Problem Does This Paper Address?

Modern systems, from IoT devices to industrial machinery, generate vast amounts of multivariate time series data – essentially, streams of readings from numerous sensors over time. Identifying anomalies (outliers or unusual patterns) in this data is crucial for system monitoring, fault detection, and preventing failures. Traditional methods often struggle because:

  • Inter-sensor dependencies are complex and dynamic: How one sensor's reading relates to another isn't static; it changes based on real-time conditions and can be influenced by unexpected events. Previous approaches often assumed fixed relationships, leading to missed anomalies or false alarms.
  • Long-term temporal dependencies are overlooked: The current state of a system can be influenced by events that happened much earlier. Capturing these long-range temporal patterns is essential for accurate prediction and anomaly detection, but many models struggle with long sequences.

This paper aims to address these limitations by proposing a model that can effectively capture both the dynamic, multiscale inter-sensor relationships and the long-term temporal dependencies in multivariate time series.

Methods

The proposed Disentangled Dynamic Deviation Transformer Network (D³TN) is composed of two main parts: a prediction module and a dynamic deviation scoring module.

The prediction module is the core of the anomaly detection system and consists of three key components:

  1. Feature Block: This block focuses on modeling inter-sensor dependencies. It uses a novel "disentangled global graph convolution layer" to capture fixed, multiscale sensor relationships based on static topology, while also modeling dynamic, real-time dependencies. A "local attention layer" then further refines this by using a self-attention mechanism to capture dynamic, directed interactions between sensors, considering their real-time monitoring values and even their spatial embeddings.
  2. Temporal Block: This component is designed to capture long-term temporal dependencies. It leverages a transformer-based self-attention mechanism to learn hidden, time-varying relationships across multiple time steps, effectively processing remote dependencies in parallel.
  3. GRU (Gated Recurrent Unit): The outputs from the feature and temporal blocks, along with the raw input data, are fed into a GRU. This recurrent neural network is used to capture sequential patterns in the fused data, further enhancing the prediction accuracy.

The dynamic deviation scoring module then uses the difference between the predicted values from the prediction module and the actual observed values to calculate an anomaly score. An automatic thresholding method (POT-MoM) is employed to dynamically determine when this score indicates an anomaly.

Datasets

The paper evaluates D³TN on three publicly available real-world datasets:

  • MSL (MSL): A spacecraft dataset.
  • SMAP (SMAP): Another spacecraft dataset.
  • SMD (SMD): An application server dataset, noted as the largest public dataset for multivariate time series anomaly detection.

These datasets are standard benchmarks in the field, each containing labeled anomalies.

Key Insights and Ideas

  • Disentangling sensor relationships is key: The paper argues that simply modeling fixed sensor correlations is insufficient. By disentangling static (topology-based) and dynamic (real-time) inter-sensor dependencies, the model can better adapt to changing system behaviors.
  • Multiscale aggregation for sensor graphs: Instead of treating sensor relationships as a single graph, the model aggregates information across multiple scales, allowing it to capture both local and more distant correlations.
  • Transformer for long-range temporal patterns: The use of transformer's self-attention mechanism is crucial for efficiently capturing dependencies that span long time intervals, overcoming limitations of traditional RNNs.
  • Combining feature and temporal insights: The synergy between the feature block (inter-sensor) and temporal block (time-dependent) is vital. The model's strength lies in its ability to jointly model these two dimensions.
  • Dynamic thresholding for robust anomaly detection: Relying on fixed thresholds can be problematic. The adaptive thresholding method ensures that the anomaly detection is sensitive to the specific data distribution.

Key Results

D³TN consistently outperforms state-of-the-art anomaly detection methods across the MSL, SMAP, and SMD datasets. Specifically, the paper reports improvements of 2.9%, 4.3%, and 0.7% over the best existing methods on MSL, SMAP, and SMD, respectively, in terms of F1-score. Ablation studies demonstrate that removing any single component (feature block, temporal block, or GRU) leads to a significant degradation in performance, highlighting the importance of each part of the D³TN architecture. The model also shows strong capabilities in anomaly explanation, helping operators pinpoint the root causes of anomalies.

Limitations and Future Work

The authors acknowledge that while the unsupervised learning approach assumes normal training data, real-world scenarios might contain anomalies in the training set, which could negatively impact performance. They suggest that pre-training, similar to techniques used in NLP and image processing, could further improve anomaly detection capabilities. Future work could also focus on adapting the model for more complex engineering applications, improving its reliability and root cause diagnosis capabilities, and exploring ways to handle potential anomalies within the training data itself.

Summary updated

BibTeX
@misc{wang2023disentangled,
  author = {Chunzhi Wang and Shaowen Xing and Rong Gao and Lingyu Yan and Naixue Xiong and Ruoxi Wang},
  title = {{Disentangled Dynamic Deviation Transformer Networks for Multivariate Time Series Anomaly Detection}},
  year = {2023},
  doi = {10.3390/s23031104},
}
Figure