EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables

Penglei Gao, Xi Yang, Rui Zhang, Ping Guo, John Y. Goulermas, Kaizhu Huang

TL;DR

EgPDE-Net is a novel continuous-time neural network that models multivariate time series as a system of Partial Differential Equations (PDEs). By mathematically reducing a complex PDE problem into a tractable, regularized Ordinary Differential Equation (ODE) framework, it enables high-accuracy forecasting at any arbitrary time point while effectively capturing how exogenous variables drive the target series.

What Problem Does This Paper Address?

Traditional deep learning models for time series, such as Recurrent Neural Networks (RNNs), are fundamentally discrete—they make predictions at fixed intervals (e.g., $T+1, T+2$). This breaks down when we need arbitrary-step prediction, such as forecasting a value at $T+1.5$ when data is only sampled at integer steps.

While "Neural ODEs" were introduced to create continuous-time models, they often struggle with multivariate data. Specifically, they rarely account for the complex inter-series correlations and time-dependencies between exogenous variables (external factors like weather) and the target series (the variable we want to predict, like electricity demand). The authors argue that these systems are better described by PDEs, which can model changes across multiple independent variables simultaneously, but PDEs are notoriously difficult to solve numerically within a neural network.

Comparison of discrete vs. continuous (arbitrary-step) prediction.

Methods

The core contribution is the Exogenous-guided Partial Differential Equation Network (EgPDE-Net). The authors propose a framework that treats the multivariate system as a reaction-diffusion PDE. To make this solvable, they introduce a mathematical "bridge" (Lemma III.1) that transforms the PDE into two coupled ODE pipelines:

  1. The Exogenous Pipeline: Exogenous variables are processed through a self-attention block to capture global and local correlations. A first ODE solver then generates "partial differential weights"—essentially a dynamic map of how these external factors should guide the target series.
  2. The Target Pipeline: The target series is processed via a GRU to capture its internal temporal patterns. A second ODE solver then generates the final latent trajectory, using the weights from the first pipeline as a regularization term to "steer" the prediction.

By integrating these ODEs, the model can produce a prediction for any continuous time $t$, not just fixed intervals.

Fig. 2. Model structure of EgPDE-Net. The hidden representations of hx0 and hy0 are generated with the self-attention block and GRU block respectively. Then the above ODE net is leveraged to obtain the latent states representing the partial derivative weights guiding the generation of the trajectory of final latent state zt in the below ODE net. t1, · · · , tM are the forecasting time points.

Datasets

The model was evaluated on four real-world multivariate datasets:

  • SML2010: Indoor temperature forecasting (13 exogenous variables).
  • Electricity: Hourly electricity consumption in the US (15 weather-related exogenous variables).
  • ETTh1 & ETTh2: Electricity Transformer Temperature (6 power load features).

To test arbitrary-step prediction, the authors re-sampled these datasets to half their original frequency during training and tasked the models with predicting the "missing" intermediate points during testing.

Key Insights and Ideas

  • PDEs as a Natural Fit: While ODEs only describe evolution over time, PDEs allow the model to learn derivatives with respect to both time and the exogenous variable space. This provides a more rigorous mathematical foundation for multivariate forecasting.
  • Guidance as Regularization: The authors treat the influence of exogenous variables as a "regularized guidance" term. This prevents the model from drifting and ensures the predicted trajectory is physically and logically consistent with external drivers.
  • Tractability via Reduction: The "aha moment" is the realization that a complex PDE can be approximated by two neural networks acting as "weak solutions." This avoids the computational nightmare of traditional numerical PDE solvers while retaining the model's continuous nature.
  • Interpretability: The use of self-attention allows researchers to see which exogenous variables (e.g., "FeelsLike" temperature vs. "Humidity") are most influential at specific times.

Key Results

EgPDE-Net consistently outperformed both discrete (LSTNet, IMV-tensor) and continuous (Latent ODE, ETN-ODE) baselines.

  • Arbitrary-Step Accuracy: On average, it reduced RMSE by 9.85% and MAE by 13.98% compared to the best-performing baseline.
  • Standard Multi-Step Prediction: Even on fixed-interval tasks ($M=1, 5, 10$), EgPDE-Net achieved the lowest errors in nearly all categories across all four datasets.
  • Efficiency: Despite the complexity of the PDE formulation, the model maintains a competitive inference time and a relatively small parameter footprint (e.g., ~50k parameters for the Electricity dataset).

Limitations and Future Work

The authors acknowledge a few areas for improvement:

  • Computational Overhead: Continuous models, by nature, require more training and inference time than simple discrete networks due to the ODE integration steps.
  • Discrete Encoding: The current encoding layers (GRU and Self-Attention) are still discrete. The authors suggest exploring fully continuous encoding networks in the future.
  • Missing Data: While the model handles arbitrary time steps, it does not yet explicitly model "informative missingness" (where the fact that data is missing carries its own meaning), which is a common challenge in clinical and sensor data.

Summary updated

Abstract

While exogenous variables have a major impact on performance improvement in time series analysis, inter-series correlation and time dependence among them are rarely considered in the present continuous methods. The dynamical systems of multivariate time series could be modelled with complex unknown partial differential equations (PDEs) which play a prominent role in many disciplines of science and engineering. In this paper, we propose a continuous-time model for arbitrary-step prediction to learn an unknown PDE system in multivariate time series whose governing equations are parameterised by self-attention and gated recurrent neural networks. The proposed model, \underline{E}xogenous-\underline{g}uided \underline{P}artial \underline{D}ifferential \underline{E}quation Network (EgPDE-Net), takes account of the relationships among the exogenous variables and their effects on the target series. Importantly, the model can be reduced into a regularised ordinary differential equation (ODE) problem with special designed regularisation guidance, which makes the PDE problem tractable to obtain numerical solutions and feasible to predict multiple future values of the target series at arbitrary time points. Extensive experiments demonstrate that our proposed model could achieve competitive accuracy over strong baselines: on average, it outperforms the best baseline by reducing $9.85\%$ on RMSE and $13.98\%$ on MAE for arbitrary-step prediction.
BibTeX
@article{gao2022egpde,
  author = {Penglei Gao and Xi Yang and Rui Zhang and Ping Guo and John Y. Goulermas and Kaizhu Huang},
  title = {{EgPDE-Net: Building Continuous Neural Networks for Time Series Prediction with Exogenous Variables}},
  journal = {IEEE Trans. Cybernetics},
  year = {2022},
  doi = {10.1109/tcyb.2024.3364186},
}
Figure