Graph-Enabled Reinforcement Learning for Time Series Forecasting With Adaptive Intelligence

Thanveer Shaik, Xiaohui Tao, Haoran Xie, Lin Li, Jianming Yong, Yuefeng Li

TL;DR

GraphRL is a novel framework that combines Temporal Graphical Convolutional Networks (T-GCN) with Reinforcement Learning (RL) to predict complex time-series data. By explicitly modeling spatial-temporal dependencies through graphs and using an RL agent to optimize decision-making, it outperforms traditional deep learning models in healthcare, traffic, and weather forecasting.

What Problem Does This Paper Address?

Traditional time-series forecasting relies heavily on models like LSTMs and RNNs. While effective, these models have two major "blind spots":

  1. Structural Rigidity: They often assume data is equally spaced and lack a natural way to incorporate the underlying "shape" or connectivity of the data (e.g., how traffic at one intersection affects another).
  2. Passive Prediction: Supervised learning models are typically passive; they predict a value but don't account for how errors accumulate over time in dynamic environments where a prediction might trigger an intervention.

The authors address these gaps by introducing GraphRL, a framework designed for "Adaptive Intelligence." It treats forecasting not just as a regression task, but as a dynamic monitoring process where an agent learns to navigate complex, graph-structured environments.

Methods

The core of the paper is the GraphRL framework, which utilizes two primary components:

  • Temporal Graphical Convolutional Networks (T-GCN): This is the "brain" of the system. It uses a GCN layer to capture spatial features (relationships between nodes in a graph) and a Gated Recurrent Unit (GRU) layer to capture temporal patterns.
  • Reinforcement Learning (DQN): The authors integrate T-GCN into a Deep Q-Network (DQN) architecture. Uniquely, they use T-GCN in two places:
  • The Predictive Environment: Unlike standard RL where the environment is static, this environment uses T-GCN to forecast future states, allowing the system to be proactive.
  • The Agent: A T-GCN model approximates the Q-function, helping the agent choose the best "action" (e.g., issuing a medical alert or changing a traffic light) based on predicted future states.

To squeeze out maximum performance, the team employed Bayesian Optimization to fine-tune critical hyperparameters like the learning rate ($\alpha$) and the discount factor ($\gamma$).

Datasets

The framework was stress-tested across three diverse domains:

  1. WESAD (Healthcare): Biometric time-series data (ECG, PPG, etc.) from 17 participants, used here for heart rate prediction and medical monitoring.
  2. LA Traffic: Real-time urban traffic counts and speeds from the Los Angeles Department of Transportation.
  3. LSTW (Weather): A large-scale dataset combining traffic conditions with weather events across the United States.

Key Insights and Ideas

  • The "Predictive Environment" Concept: One of the most interesting "aha" moments is the shift from a reactive environment to a predictive one. By embedding a forecasting model inside the RL environment, the agent receives rewards based on its ability to handle future states, not just the current one.
  • Graph-Awareness Matters: By treating sensors (vitals, traffic loops) as nodes in a graph, the model captures "neighborhood" effects that LSTMs miss. If a patient's heart rate and respiration are both trending poorly, the graph structure helps the model understand their combined physiological impact more naturally.
  • Discount Factor Sensitivity: Through Bayesian optimization, the authors found that a lower discount factor ($\gamma$) often performed better. This suggests that in high-stakes forecasting (like healthcare), immediate accuracy in the next few time steps is often more valuable than distant future rewards.

Key Results

GraphRL consistently outperformed state-of-the-art baselines like ELMA and standard GRU models across all datasets:

  • Accuracy: In healthcare forecasting (15-minute window), GraphRL achieved a Mean Absolute Error (MAE) of 0.56, significantly lower than the GRU baseline of 0.95.
  • RL Performance: When compared to other RL agents (like PPO, A2C, or DDPG), the GraphRL agent achieved the highest cumulative rewards. For example, on the WESAD dataset, GraphRL scored 48,790, while the standard DQN scored 41,986.
  • Robustness: The model maintained superior performance across various forecasting horizons (15, 30, 45, and 60 minutes), proving its stability over time.

Limitations and Future Work

While powerful, the authors acknowledge a few hurdles:

  • Computational Intensity: Integrating T-GCNs with RL is computationally expensive, which may pose challenges for scaling to massive, city-wide graphs in real-time.
  • Data Dependency: The framework's efficacy is highly dependent on the quality and structure of the input graph. Irregular or poorly defined graph edges could degrade performance.
  • Future Directions: The authors plan to incorporate more complex spatial and geographical dimensions (like 3D terrain for weather) and explore real-time adaptive learning strategies to help the model pivot even faster when environmental patterns shift abruptly.

Summary updated

Abstract

Reinforcement learning (RL) is renowned for its proficiency in modeling sequential tasks and adaptively learning latent data patterns. Deep learning models have been extensively explored and adopted in regression and classification tasks. However, deep learning has limitations, such as the assumption of equally spaced and ordered data, and the inability to incorporate graph structure in time-series prediction. Graph Neural Network (GNN) can overcome these challenges by capturing the temporal dependencies in time-series data effectively. In this study, we propose a novel approach for predicting time-series data using GNN, augmented with Reinforcement Learning(GraphRL) for monitoring. GNNs explicitly integrate the graph structure of the data into the model, enabling them to naturally capture temporal dependencies. This approach facilitates more accurate predictions in complex temporal structures, as encountered in healthcare, traffic, and weather forecasting domains. We further enhance our GraphRL model's performance through fine-tuning with a Bayesian optimization technique. The proposed framework surpasses baseline models in time-series forecasting and monitoring. This study's contributions include introducing a novel GraphRL framework for time-series prediction and demonstrating GNNs' efficacy compared to traditional deep learning models, such as Recurrent Neural Networks (RNN) and Long Short-Term Memory Networks(LSTM). Overall, this study underscores the potential of GraphRL in yielding accurate and efficient predictions within dynamic RL environments.
BibTeX
@article{shaik2024graph,
  author = {Thanveer Shaik and Xiaohui Tao and Haoran Xie and Lin Li and Jianming Yong and Yuefeng Li},
  title = {{Graph-Enabled Reinforcement Learning for Time Series Forecasting With Adaptive Intelligence}},
  journal = {IEEE Transactions on Emerging Topics in Computational Intelligence},
  year = {2024},
  url = {https://openreview.net/forum?id=XnbZanH6pr},
}
Figure