Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection

Cristian Challu, Peihong Jiang, Ying Nian Wu, Laurent Callot

Detecting anomalies in multivariate time series is a high-stakes challenge, critical for everything from monitoring server clusters to ensuring the safety of spacecraft. While Deep Learning has pushed the boundaries of what's possible, many current state-of-the-art (SoTA) models are computationally expensive, rely on complex Recurrent Neural Network (RNN) architectures, and buckle when faced with the "messy" data typical of real-world IoT sensors—such as missing values or variable features.

A new paper from researchers at CMU and Amazon Research introduces DGHL (Deep Generative model with Hierarchical Latent factors), a model that tackles these issues head-on. By rethinking how we represent time-series dynamics in latent space, the authors have created a model that is not only more accurate but also significantly faster and more robust than its predecessors.

TL;DR

DGHL is a novel generative model for time-series anomaly detection that uses a hierarchical latent space and a top-down convolutional generator. It achieves state-of-the-art performance while training up to 10x faster than RNN-based models and remains remarkably accurate even when 90% of the input data is missing.

What Problem Does This Paper Address?

Most reconstruction-based anomaly detection models (like VAEs or GANs) try to learn the "normal" state of a system. If a new data point can't be reconstructed well, it's flagged as an anomaly. However, these models face three main hurdles:

  1. Temporal Complexity: Modeling long-term dependencies usually requires RNNs, which are slow to train because they process data sequentially.
  2. Rigid Dynamics: Many models assume the underlying dynamics of a system are constant over time, which is rarely true in the real world.
  3. Data Imperfection: Real-world sensors often fail, leading to missing values. Most models require "perfect" data or complex imputation steps to function.

Methods

DGHL introduces two key innovations to solve these problems:

1. Hierarchical Latent Factors

Instead of mapping a window of time-series data to a single latent vector, DGHL uses a hierarchy. As shown in Figure 1, latent vectors at higher levels are shared across multiple sub-windows (capturing long-term trends and seasonalities), while lower-level vectors are specific to individual sub-windows (capturing local fluctuations).

Figure 1: Example of hierarchical latent factor space for a = [1, 3, 6]. On each level l, the latent vectors of al consecutive sub-windows are tied. For instance, the latent vector on the highest layer, L, is shared by all sub-windows of Y .

2. Top-Down ConvNet & Alternating Back-Propagation

Rather than using an RNN, DGHL uses a top-down Convolutional Network (ConvNet) as its generator (Figure 2). This allows for massive parallelization during training.

Crucially, the model does not use an "encoder" network. Instead, it uses Alternating Back-Propagation (ABP). During training, it uses Langevin Dynamics (a form of MCMC sampling) to infer the latent vectors that best explain the observed data, then updates the generator's weights. This "explain-away" process allows the model to naturally handle missing data by simply ignoring the missing dimensions during the latent inference step.

Figure 2: Generator architecture, which maps a latent vector Z to a time-series window. Each layer of the ConvNet increases the temporal dimension and reduces the filters by a factor of 2. A batch normalization layer and ReLU activations are included between each convolutional layer.

Datasets

The authors evaluated DGHL on four popular multivariate time-series benchmarks:

  • SMD (Server Machine Dataset): 5 weeks of data from 28 server machines.
  • SMAP & MSL: Real telemetric data from NASA satellites and the Mars rover.
  • SWaT (Secure Water Treatment): Data from a water treatment testbed used for cyber-security research.

They also performed novel occlusion experiments on the SMD dataset, where they intentionally masked large portions of the data to test the model's robustness.

Key Insights and Ideas

  • Speed without Sacrifice: By replacing sequential RNNs with a ConvNet, DGHL achieves a 10x speedup in training time compared to models like OmniAnomaly, without losing the ability to model complex temporal relations.
  • Inherent Robustness: Because the model infers latents directly from whatever data is available, it is natively robust to missing values. It doesn't need a separate imputation model.
  • Non-Parametric Dynamics: The hierarchical structure allows the model to share information across time windows to capture long-term dynamics without assuming those dynamics are constant, making it more flexible for real-world applications.

Key Results

DGHL consistently achieved Top-2 F1 scores across all tested datasets, often outperforming complex SoTA models like MTAD-GAT and THOC.

  • SMD Performance: DGHL reached an F1 score of 86.18, significantly higher than the next best reconstruction-based model (NCAD at 80.84).
  • Extreme Robustness: In occlusion experiments (Figure 5), DGHL maintained an F1 score of ~76 even with 90% missing data, whereas competing models like OmniAnomaly saw their performance plummet to near-baseline levels.

Figure 4: F1 scores on SMD dataset Figure 5: Robustness to data occlusion

Limitations and Future Work

The authors note that DGHL currently assumes constant variance for each feature. While they use normalization to mitigate this, extending the model to learn the variance of each feature dynamically could improve performance further.

Additionally, they acknowledge the ongoing debate in the community regarding the quality of standard benchmarks (some may contain mislabeled anomalies). They suggest that creating more relevant datasets with "contextual" anomalies—which are hard even for humans to spot—is a pressing necessity for the field.

For those interested in implementation, the authors have made their code available at https://github.com/cchallu/dghl.

Summary updated

Abstract

Multivariate time series anomaly detection has become an active area of research in recent years, with Deep Learning models outperforming previous approaches on benchmark datasets. Among reconstruction-based models, most previous work has focused on Variational Autoencoders and Generative Adversarial Networks. This work presents DGHL, a new family of generative models for time series anomaly detection, trained by maximizing the observed likelihood by posterior sampling and alternating back-propagation. A top-down Convolution Network maps a novel hierarchical latent space to time series windows, exploiting temporal dynamics to encode information efficiently. Despite relying on posterior sampling, it is computationally more efficient than current approaches, with up to 10x shorter training times than RNN based models. Our method outperformed current state-of-the-art models on four popular benchmark datasets. Finally, DGHL is robust to variable features between entities and accurate even with large proportions of missing values, settings with increasing relevance with the advent of IoT. We demonstrate the superior robustness of DGHL with novel occlusion experiments in this literature. Our code is available at https://github.com/cchallu/dghl.
BibTeX
@inproceedings{challu2022deep,
  author = {Cristian Challu and Peihong Jiang and Ying Nian Wu and Laurent Callot},
  title = {{Deep Generative model with Hierarchical Latent Factors for Time Series Anomaly Detection}},
  booktitle = {International Conference on Artificial Intelligence and Statistics},
  year = {2022},
}
Figure