Explainable Tensorized Neural Ordinary Differential Equations for Arbitrary-Step Time Series Prediction

Penglei Gao, Xi Yang, Rui Zhang, Kaizhu Huang, John Yannis Goulermas

TL;DR

This paper introduces ETN-ODE, a novel continuous neural network for multivariate time series prediction that can forecast values at arbitrary future time points, not just at fixed intervals. By combining a Tensorized GRU with an attention mechanism and Ordinary Differential Equations (ODEs), ETN-ODE achieves state-of-the-art performance and offers interpretability into how different variables and time steps contribute to predictions.

What Problem Does This Paper Address?

Predicting future values of time series data is crucial across many domains, from finance to weather forecasting. Traditional methods often struggle with multivariate time series (where multiple related variables are involved) and are limited to predicting at fixed, discrete time steps that align with the sampling rate of the input data. This limitation is problematic in real-world scenarios where predictions are needed at arbitrary future moments (e.g., predicting electricity consumption in 15 minutes when data is sampled hourly). Furthermore, many deep learning models for time series prediction are "black boxes," making it difficult to understand why they make certain predictions, which is a significant drawback in critical applications. This paper aims to bridge this gap by developing a model that is both accurate for arbitrary-step, multivariate time series prediction and provides interpretable insights.

Methods

The core of the ETN-ODE framework is its ability to model continuous-time dynamics. This is achieved by:

  • Tensorized GRU (TGRU): Instead of a standard GRU that uses a vector for its hidden state, the TGRU uses a matrix. This allows it to learn independent representations for each individual input feature in a multivariate time series, capturing their unique dynamics. This is more parameter-efficient than a Tensorized LSTM.
  • Tandem Attention: This mechanism comprises two parts:
  • Temporal Attention: Focuses on the importance of different historical time steps for the prediction.
  • Variable Attention: Determines the contribution of each exogenous variable to the target series. Together, these attention mechanisms provide interpretability by highlighting which past time points and which input variables are most influential.

  • Ordinary Differential Equations (ODEs): The TGRU's hidden states, enhanced by the tandem attention, are fed into an ODE network. The ODE network models the derivative of the latent states, allowing for continuous-time prediction. This means that once the model is trained, it can generate predictions for any future time point, not just those corresponding to the input data's sampling frequency. The ODE network's parameters are learned using a neural network.

Datasets

The paper evaluates ETN-ODE on four real-world datasets:

  • SML2010: A dataset related to room temperature and other environmental factors.
  • Energy: Likely related to energy consumption.
  • PM2.5: Air quality data.
  • Nasdaq100: Stock market data.

These datasets are used for both standard multi-step prediction and the novel arbitrary-step prediction tasks. No new benchmarks are introduced; rather, existing datasets are utilized to demonstrate the model's capabilities.

Key Insights and Ideas

  • Continuous-time prediction is powerful: The ODE framework fundamentally changes how time series are modeled, moving beyond discrete steps to a continuous representation. This is crucial for applications requiring predictions at arbitrary future moments.
  • Multivariate dynamics require specialized handling: A simple concatenation of features isn't enough. The TGRU's matrix-based hidden states are key to capturing the distinct patterns within each variable of a multivariate series.
  • Interpretability is achievable: The tandem attention mechanism is not just for accuracy; it provides concrete insights into why a prediction is made by showing the relative importance of different time steps and variables. This is a significant advantage over many black-box deep learning models.
  • TGRU offers a sweet spot: The paper suggests that TGRU, with its fewer parameters compared to Tensorized LSTM, provides a good balance of performance and efficiency for multivariate time series.

Key Results

  • Arbitrary-step Prediction: ETN-ODE significantly outperforms the Latent-ODE baseline on arbitrary-step prediction tasks across multiple datasets, achieving lower RMSE and MAE. The model demonstrates consistent performance for both integer and fractional time steps.
  • Standard Multi-step Prediction: ETN-ODE achieves state-of-the-art results on standard multi-step prediction tasks, outperforming several deep learning baselines (Retain, UA, IMV-tensor, TLSTM) in terms of RMSE and MAE. It shows particular strength in capturing long-term dependencies.
  • Interpretability: Visualizations of the attention weights clearly show how the model focuses on specific variables (e.g., temperature, CO2) and time periods, aligning with real-world intuition.

Limitations and Future Work

  • Computational Cost: While the TGRU is more efficient than Tensorized LSTM, ODE-based models can still be computationally intensive, especially during training.
  • Data Nature: The paper notes that for highly complex data like the Energy dataset, both models (ETN-ODE and Latent-ODE) experience larger errors, suggesting that extreme data complexity can still pose a challenge.
  • Future Work: The paper implicitly suggests that further exploration of ODE-based models for time series, potentially with more advanced attention mechanisms or different ODE formulations, could yield even better results. The focus on interpretability also opens doors for more sophisticated explainability tools.

Summary updated

Abstract

We propose a continuous neural network architecture, termed Explainable Tensorized Neural Ordinary Differential Equations (ETN-ODE), for multi-step time series prediction at arbitrary time points. Unlike the existing approaches, which mainly handle univariate time series for multi-step prediction or multivariate time series for single-step prediction, ETN-ODE could model multivariate time series for arbitrary-step prediction. In addition, it enjoys a tandem attention, w.r.t. temporal attention and variable attention, being able to provide explainable insights into the data. Specifically, ETN-ODE combines an explainable Tensorized Gated Recurrent Unit (Tensorized GRU or TGRU) with Ordinary Differential Equations (ODE). The derivative of the latent states is parameterized with a neural network. This continuous-time ODE network enables a multi-step prediction at arbitrary time points. We quantitatively and qualitatively demonstrate the effectiveness and the interpretability of ETN-ODE on five different multi-step prediction tasks and one arbitrary-step prediction task. Extensive experiments show that ETN-ODE can lead to accurate predictions at arbitrary time points while attaining best performance against the baseline methods in standard multi-step time series prediction.
BibTeX
@article{gao2023explainable,
  author = {Penglei Gao and Xi Yang and Rui Zhang and Kaizhu Huang and John Yannis Goulermas},
  title = {{Explainable Tensorized Neural Ordinary Differential Equations for Arbitrary-Step Time Series Prediction}},
  journal = {IEEE Trans. Knowl. Data Eng.},
  year = {2023},
  doi = {10.1109/tkde.2022.3167536},
}
Figure