Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation

Tryambak Gangopadhyay, Sin Yong Tan, Zhanhong Jiang, Rui Meng, Soumik Sarkar

TL;DR

The authors introduce STAM (Spatiotemporal Attention Mechanism), a novel deep learning architecture for multivariate time series that simultaneously identifies when (temporal) and which (spatial) variables are most important for a prediction. Unlike previous state-of-the-art models, STAM is strictly causal, scales efficiently to many variables, and provides interpretability that aligns directly with domain knowledge.

What Problem Does This Paper Address?

Multivariate time series (MTS) data—where multiple variables are tracked over time—is foundational to fields like healthcare, finance, and climate science. While models like LSTMs are great at capturing long-term dependencies, they are often "black boxes."

Existing "interpretable" models face three major hurdles:

  1. Causality: Many models (like DA-RNN) inadvertently use "future" information during the encoding phase to calculate attention weights, making them unsuitable for real-time deployment.
  2. Scalability: Hierarchical models often become computationally intractable as the number of input variables increases.
  3. Direct Interpretability: Often, spatial (variable-wise) attention is buried deep in the encoder, making it hard to see how a specific variable directly influences the final output.

Methods

The researchers propose the Spatiotemporal Attention Mechanism (STAM). Its architecture is built on a few key innovations:

  • Dual Embeddings: The model generates temporal embeddings using a two-layer LSTM encoder and spatial embeddings using independent feed-forward neural networks for each feature. This independence is what allows the model to scale well.
  • Parallel Decoder Attentions: In the decoder phase, STAM uses two separate attention mechanisms. One "soft searches" for the most relevant time steps, while the other searches for the most significant variables.
  • Direct Alignment: Crucially, both spatial and temporal context vectors are aligned directly with the output variable. This ensures that the attention weights actually represent the contribution of an input to the prediction.
  • STAM-Lite: The authors also proposed a "Lite" version that uses a single LSTM in the decoder to process a concatenated context vector, offering a faster alternative with slightly less complexity.

Datasets

The model was evaluated on three distinct real-world datasets:

  1. Beijing PM2.5 (Pollution): Predicting air pollution levels based on eight meteorological variables (dew point, wind speed, etc.).
  2. Building HVAC: A domain-specific dataset from an air handling unit, using nine variables to predict average zone temperature.
  3. EHR (MIMIC-III): Clinical data from ICU stays, using seven variables (Heart Rate, Glucose, etc.) to predict a patient's respiratory rate.

Key Insights and Ideas

  • Causality Matters: By ensuring the model only depends on past inputs, STAM is more robust for real-world forecasting where the future is unknown.
  • The "Where" and the "When": Most models focus on temporal attention (which hour was important?). STAM places equal weight on spatial attention (which sensor was important?), providing a 2D map of importance.
  • Domain Knowledge Validation: The authors didn't just look at error metrics; they checked if the model's "logic" matched physics and medicine. For example, in the pollution dataset, the model correctly prioritized "Dew Point" over "Temperature," which aligns with environmental science regarding how moisture affects particulate matter.
  • Breaking the Trade-off: Usually, adding interpretability layers slows down a model or hurts accuracy. STAM manages to maintain (and sometimes exceed) state-of-the-art accuracy while remaining computationally tractable.

Key Results

  • Performance: STAM consistently outperformed or matched baseline models (SVR, Encoder-Decoder, DA-RNN) across all datasets. For the Building dataset, STAM achieved an $R^2$ score of 0.9975, slightly edging out the complex DA-RNN.
  • Superior Interpretability: In the Building HVAC test, the baseline DA-RNN failed to distinguish between variables, assigning roughly equal importance to all. STAM, however, clearly identified the "Outside Air Damper Command" and "Discharge Air Temperature" as the primary drivers, which matches the physical reality of HVAC systems.
  • Scalability: The complexity analysis shows that STAM's inference time scales linearly with the number of variables ($N$), making it much more practical for high-dimensional sensor data than hierarchical alternatives.

Limitations and Future Work

  • Data Hunger: The authors acknowledge that capturing these complex spatiotemporal correlations effectively requires a substantial amount of training data.
  • Univariate Output: The current iteration of STAM is designed for univariate prediction (predicting one target variable). The authors suggest that extending the architecture to handle multivariate output (predicting multiple targets simultaneously) is a key direction for future research.
  • Prominence of Relationships: The effectiveness of the attention mechanism depends on how prominent the underlying relationships are within the dataset; very noisy data might still obscure interpretability.

Summary updated

Abstract

Multivariate time series modeling and prediction problems are abundant in many machine learning application domains. Accurate interpretation of the prediction outcomes from the model can significantly benefit the domain experts. In addition to isolating the important time-steps, spatial interpretation is also critical to understand the contributions of different variables on the model output. We propose a novel deep learning architecture, called spatiotemporal attention mechanism (STAM) for simultaneous learning of the most important time steps and variables. STAM is a causal (i.e., only depends on past inputs and does not use future inputs) and scalable (i.e., scales well with an increase in the number of variables) approach that is comparable to the state-of-the-art models in terms of computational tractability. We demonstrate our models’ performance on a popular public dataset and a domain-specific dataset, where the learned attention weights are validated from a domain knowledge perspective. When compared with the baseline models, the results show that STAM maintains state-of-the-art prediction accuracy while offering the benefit of accurate spatiotemporal interpretability.
BibTeX
@inproceedings{gangopadhyay2021spatiotemporal,
  author = {Tryambak Gangopadhyay and Sin Yong Tan and Zhanhong Jiang and Rui Meng and Soumik Sarkar},
  title = {{Spatiotemporal Attention for Multivariate Time Series Prediction and Interpretation}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  url = {https://openreview.net/forum?id=Ui8IbP2CfJ-},
}

Loading related content...

Figure