Addressing partial observability in reinforcement learning for energy management

Marco Biemann, Xiufeng Liu, Yifeng Zeng, Lizhen HuangBuildSys 2021 · DOI

TL;DR

This paper tackles the challenge of managing energy systems with reinforcement learning (RL) when not all relevant information is directly observable. They propose two novel approaches using recurrent neural networks and variational inference to learn policies that account for past observations, leading to significant energy savings and improved temperature control in simulated data center HVAC systems.

What Problem Does This Paper Address?

Modern energy systems, like HVAC in data centers, are complex and influenced by many factors such as weather, energy prices, and user demand. While reinforcement learning (RL) has shown promise in automating control for these systems, most existing RL approaches rely on the "Markov assumption." This assumption means the agent only considers the current state of the environment to make decisions. However, in many real-world scenarios, including energy management, this assumption is inaccurate because past information provides crucial context for optimal decision-making. For instance, understanding the recent trend of outdoor temperature is vital for predicting future heating or cooling needs.

Furthermore, directly measuring all relevant factors (like server temperatures, airflow dynamics, and building material thermodynamics) is often impractical, costly, or raises privacy concerns. This leads to a "partially observable" environment, where the agent only receives incomplete information. These problems can be framed as Partially Observable Markov Decision Processes (POMDPs). This paper aims to develop RL methods that can effectively learn and operate in such partially observable environments, going beyond the limitations of traditional Markovian approaches.

Methods

The authors propose two main methods to address partial observability in RL for energy management, both building upon actor-critic RL algorithms (specifically Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO)).

  1. Recurrent Neural Networks (RNNs) / LSTMs: This approach modifies the standard feed-forward neural network architectures used for the actor and critic in RL algorithms. Instead, they use recurrent neural networks, specifically Long Short-Term Memory (LSTM) cells. The LSTM processes sequences of past observations and actions, encoding this history into a "recurrent state." This recurrent state then serves as the input for the actor and critic, allowing the agent to implicitly learn from past information without needing to explicitly model the full state. This method is relatively straightforward to implement by replacing feed-forward layers with LSTMs.
  2. Variational Inference for Latent State Representation: This method aims to explicitly learn a representation of the hidden state of the environment. It formulates the problem as a POMDP and uses variational inference to approximate the posterior distribution of the hidden state given the observed history. The core idea is to learn a "belief state" that summarizes all relevant past information. They employ a generative model (encoder-decoder architecture, inspired by Variational Autoencoders - VAEs) to infer a latent state representation. This latent state is then used by the actor and critic for decision-making. This approach allows the agent to learn a more robust representation of the environment's state, which can be beneficial in non-stationary conditions.

Datasets

The paper evaluates their methods on a simulated data center HVAC control task. This simulation was implemented using EnergyPlus, a well-established building energy simulation software. The RL environment was previously developed by Moriyama et al. [26] and has been used in prior research [4, 24, 39]. The observation space includes outdoor air temperature, indoor temperatures of two zones, server electricity demand, and HVAC power. Actions involve adjusting temperature setpoints and airflow rates.

Key Insights and Ideas

  • History Matters: The core insight is that for energy management tasks, relying solely on the current state (Markov assumption) is suboptimal. Incorporating historical information is crucial for better control and efficiency.
  • POMDPs are a Natural Fit: Framing energy management problems as POMDPs provides a more accurate theoretical basis and guides the development of more effective RL agents.
  • RNNs/LSTMs Offer a Practical Solution: Using LSTMs is a relatively simple yet effective way to inject historical context into RL agents, leading to significant improvements over standard feed-forward architectures.
  • Latent State Learning Enhances Robustness: Learning an explicit latent state representation through variational inference can lead to more robust policies, especially in environments with changing dynamics.
  • Off-Policy Methods Excel in Data Efficiency: The paper highlights that off-policy methods (like SAC and SLAC) tend to be more data-efficient, reaching good performance levels faster than on-policy methods.

Key Results

The proposed methods, particularly SLAC (Stochastic Latent Actor-Critic, which uses variational inference) and PPO-LSTM, demonstrated significant improvements over traditional PPO and a baseline EnergyPlus controller:

  • Energy Consumption Reduction: After one month of training, SLAC and SAC-LSTM significantly reduced energy consumption compared to the baseline. After 20 years of training (simulated), PPO-LSTM achieved a 7.8% reduction compared to traditional PPO, and SLAC achieved a 5.2% reduction compared to SAC.
  • Temperature Control: All algorithms (except standard PPO) could maintain temperatures within the desired range after extensive training. SLAC showed particularly good performance in this regard.
  • Data Efficiency: The non-Markovian approaches, especially SLAC, showed considerable improvements in data efficiency compared to traditional PPO.

Limitations and Future Work

  • Computational Cost: The paper notes that using LSTMs for off-policy methods can be computationally expensive, often requiring three LSTMs for better performance.
  • Hyperparameter Sensitivity: SAC-LSTM was found to be sensitive to hyperparameters and prone to catastrophic forgetting, suggesting a need for careful tuning.
  • Simulation vs. Real World: The experiments were conducted in a simulated environment. While EnergyPlus is a robust simulator, real-world deployment might introduce additional complexities.

Future Work: The authors suggest investigating whether non-Markovian policies can achieve competitive results when provided with explicit weather or price forecasts as input. They also see a natural extension of these methods to model-based RL algorithms, as they share similar underlying concepts. Further research into combining these techniques with imitation learning is also proposed to accelerate real-world RL controller training.

Summary updated

BibTeX
@inproceedings{biemann2021addressing,
  author = {Marco Biemann and Xiufeng Liu and Yifeng Zeng and Lizhen Huang},
  title = {{Addressing partial observability in reinforcement learning for energy management}},
  booktitle = {BuildSys},
  year = {2021},
  doi = {10.1145/3486611.3488730},
}
Figure