How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system

Yangyang Fu, Shichao Xu, Qi Zhu, Zheng O’Neill, Veronica AdetolaEnergy 2023 · DOI

Abstract

This study provides a rigorous head-to-head comparison between Model Predictive Control (MPC) and Deep Reinforcement Learning (DRL) for building energy management. By benchmarking both against a theoretically "true" optimal solution (OPT), the researchers evaluate how closely these methods approximate optimality in a load-shifting context. Results indicate that while specific DRL agents like QRDQN can achieve up to 96.5% optimality—surpassing the best MPC at 90.1%—the average performance of DRL remains comparable to well-tuned MPC due to its stochastic nature. The study utilizes a containerized framework to ensure a fair, reproducible comparison across different control paradigms.

Introduction

Building HVAC systems are massive energy consumers, and "load shifting"—moving energy use to off-peak hours when electricity is cheaper—is a primary goal for modern grid integration. Two heavyweights dominate the advanced control landscape: Model Predictive Control (MPC), which uses a physics-based model to optimize decisions over a future horizon, and Deep Reinforcement Learning (DRL), which learns a control policy through trial and error.

Despite their prevalence, a "fair" comparison between these two has been historically difficult. Most studies suffer from tuning bias or lack a common benchmark for "true" optimality. This research, conducted by a multi-institutional team from Texas A&M, Northwestern University, and PNNL, seeks to level the playing field using a standardized numerical framework and automated hyperparameter tuning.

Taxonomy of Control Strategies

To compare these methods, it is essential to understand their theoretical roots. Both MPC and DRL can be viewed as branches of Approximate Dynamic Programming (ADP). As shown in the taxonomy below, DRL algorithms typically fall under value, policy, or combined approximation, while MPC variants can be categorized as implicit or explicit approximations within the same framework.

Taxonomy of control algorithms within the Approximate Dynamic Programming framework.

Methodology

The authors developed a containerized framework using Docker and Functional Mock-up Units (FMUs). This allows for a modular "plug-and-play" environment where different controllers can be tested against the same building model (case900FF from the ASHRAE BESTEST suite) under Chicago weather and Time-of-Use (TOU) pricing.

Bridging the Gap

A key challenge in this comparison is the differing terminology used in the MPC and RL communities. To ensure a fair evaluation, the researchers established a direct mapping between the two fields, treating the MPC "Controller" as the RL "Agent" and the physics-based "Plant" as the RL "Environment."

Mapping of terminology between Model Predictive Control (MPC) and Reinforcement Learning (RL).

Experimental Setup

  • The Benchmark (OPT): A primal-dual interior-point method was used to solve the entire control problem as a single nonlinear optimization, representing 100% optimality.
  • MPC: Tested with prediction horizons ($H$) ranging from 1 to 24 hours.
  • DRL: Evaluated four popular algorithms: DDQN, QRDQN (discrete), SAC (continuous), and PPO (on-policy).
  • Fairness: Ray/Tune (AutoML) was employed to automatically optimize hyperparameters for both MPC and DRL, reducing human bias.

Key Insights and Results

Optimality and Performance

The study revealed that the "perfect" MPC is not always perfect. Even with a perfect model, MPC is limited by its online optimizer, which can get stuck in local minima. In contrast, the best DRL agent (QRDQN) achieved 96.54% optimality, surpassing the best MPC (90.11%). However, when looking at average performance across multiple seeds, DRL (~90.42%) was nearly identical to a well-tuned MPC.

The Importance of Horizon

For load shifting, the prediction horizon is critical. Short horizons (e.g., 1 hour) result in reactive strategies. A long horizon (24 hours) allows for "wide" strategies, such as pre-cooling the building's thermal mass during the night to avoid expensive afternoon peak prices. MPC performance scaled significantly with horizon length.

Computational and Learning Trade-offs

  • Sample Efficiency: SAC learned the fastest, reaching MPC-level performance in just a few episodes. PPO struggled, failing to learn effective pre-cooling strategies within the training budget.
  • Computation: MPC requires significant online computation at every step, whereas DRL is computationally intensive during offline training but nearly instantaneous during deployment.

Conclusion

While DRL has the potential to outperform MPC in peak optimality, its stochastic nature means that in practice, a well-tuned MPC remains a formidable and reliable competitor. The researchers have open-sourced their library, FMU-DRL-DOCKER, to serve as a reproducible benchmark for the community. Future work will explore how these results hold up under modeling errors, unpredictable weather, and more complex multi-zone building environments.

Summary updated

BibTeX
@inproceedings{fu2023how,
  author = {Yangyang Fu and Shichao Xu and Qi Zhu and Zheng O’Neill and Veronica Adetola},
  title = {{How good are learning-based control v.s. model-based control for load shifting? Investigations on a single zone building energy system}},
  booktitle = {Energy},
  year = {2023},
  doi = {10.1016/j.energy.2023.127073},
}
Figure