Abstract
Multivariate time series modeling and prediction problems are abundant in many machine learning application domains. Accurate interpretation of such prediction outcomes from a machine learning model that explicitly captures temporal correlations can significantly benefit the domain experts. In this context, temporal attention has been successfully applied to isolate the important time steps for the input time series. However, in multivariate time series problems, spatial interpretation is also critical to understand the contributions of different variables on the model outputs. We propose a novel deep learning architecture, called spatiotemporal attention mechanism (STAM) for simultaneous learning of the most important time steps and variables. STAM is a causal (i.e., only depends on past inputs and does not use future inputs) and scalable (i.e., scales well with an increase in the number of variables) approach that is comparable to the state-of-the-art models in terms of computational tractability. We demonstrate our models' performance on two popular public datasets and a domain-specific dataset. When compared with the baseline models, the results show that STAM maintains state-of-the-art prediction accuracy while offering the benefit of accurate spatiotemporal interpretability. The learned attention weights are validated from a domain knowledge perspective for these real-world datasets.
Introduction
In fields such as healthcare, finance, and engineering, multivariate time series data—where multiple variables evolve over time—is ubiquitous. While Long Short-Term Memory (LSTM) networks are effective at predicting these sequences, they often function as "black boxes," providing little insight into their decision-making process. Previous attempts to use attention mechanisms to highlight important time steps (temporal attention) or variables (spatial attention) have faced significant hurdles:
- Causality: Many models utilize information from the entire input sequence, effectively using "future" information to explain a specific point in time.
- Interpretability Gap: Spatial attention is often buried deep within the encoder, obscuring how specific variables directly impact the final output.
- Scalability: Computational complexity often grows prohibitively with the number of variables.
Methods
The Spatiotemporal Attention Mechanism (STAM) addresses these challenges through a novel dual-attention decoder architecture that maintains strict causality and scalability.
The architecture consists of three primary components:
- Temporal Encoder: Two stacked LSTM layers create temporal embeddings of the input data.
- Spatial Embedding Module: In parallel, a feed-forward neural network generates distinct spatial embeddings for each input variable.
- STAM Decoder: This component utilizes two separate LSTMs ($LSTM_G$ for spatial and $LSTM_S$ for temporal). The spatial attention mechanism identifies the most influential variables, while the temporal attention mechanism isolates relevant past time steps. Crucially, both are aligned directly with the output variable to ensure clear interpretability.
A simplified version, STAM-Lite, was also developed to reduce computational complexity by using a single LSTM in the decoder while preserving interpretability.
Experimental Evaluation
STAM was evaluated on three distinct real-world datasets to prove its versatility:
- Beijing PM2.5 (Pollution): Predicting hourly air pollution levels based on meteorological data (dew point, wind speed, etc.).
- Building HVAC: Data from an air handling unit in a cooling system. The goal was to predict average zone temperature based on nine variables like damper commands and fan speeds.
- EHR (MIMIC-III): A massive electronic health record database used to predict a patient's respiratory rate based on seven clinical variables over a 24-hour window.
Key Results
STAM consistently achieved competitive or superior results across all metrics (RMSE, MAE, and $R^2$):
- Pollution Dataset: STAM achieved a Mean Absolute Error (MAE) of 29.853, outperforming the popular DA-RNN (31.267).
- Building Dataset: STAM hit an $R^2$ score of 0.9975, matching the best baseline models while providing much clearer variable importance.
- EHR Dataset: STAM achieved the lowest RMSE (4.5913) among all tested models.
- Efficiency: Despite its dual-path decoder, STAM remained computationally tractable, with training times comparable to much simpler attention models.
Discussion and Conclusion
STAM demonstrates that high-performance prediction and human-readable interpretation are not mutually exclusive. By maintaining strict causality, it is suitable for real-time applications such as hospital monitoring or power grid management. Furthermore, the attention weights generated by the model align with physical laws and domain expertise, such as identifying "Discharge Air Temperature" as a primary driver in HVAC systems. Future work will explore extending STAM to multi-output problems and testing its robustness on more diverse, noisy datasets.
Summary updated