TL;DR
This paper decouples the choice of control algorithm (MPC vs. RL) from the mathematical formulation of thermal comfort (hard vs. soft constraints) in building HVAC systems. The researchers found that the way comfort is framed and weighted impacts energy performance far more significantly than the choice of the algorithm itself.
What Problem Does This Paper Address?
Building climate control is a classic multi-objective optimization problem: how do we minimize electricity use while keeping indoor temperatures within a "comfort zone"? Traditionally, researchers have approached this in two ways:
- Hard Constraints: The controller is strictly forbidden from violating temperature bounds.
- Soft Constraints: Violations are allowed but penalized in the cost function using a weight ($w$).
While many studies compare Model Predictive Control (MPC) and Reinforcement Learning (RL), they often use different comfort formulations for each. This makes it impossible to tell if a performance gap is due to the algorithm's intelligence or simply how "strictly" it was told to prioritize comfort. This paper isolates these variables to provide a fair "apples-to-apples" comparison.
Methods
The authors evaluated three primary algorithms: Model Predictive Control (MPC), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG).
For each algorithm, they implemented:
- Soft Formulations: A weighted sum of energy and comfort violations. They tested a grid of twelve weights ($w$) to map out the trade-off.
- Hard Formulations:
- For MPC, they used an "exact penalty" method (a very high weight that mimics a hard constraint).
- For RL, they used Lagrangian methods (SAC-Lagrangian and primal-dual DDPG), where the penalty weight is automatically adapted during training to satisfy a constraint on average.
The control loop for these systems is visualized in Figure 1, showing how the controller interacts with the heat pump and floor heating system based on measurements and forecasts.
Datasets
The study utilized the BOPTEST (Building Optimization Testing) framework, specifically the bestest_hydronic_heat_pump case. This is a high-fidelity emulator of a single-zone residential building in Brussels equipped with a 15 kW air-to-water heat pump. The simulation includes a full year of realistic disturbance data, including weather and occupancy patterns for a family of five.
Key Insights and Ideas
- Formulation Trumps Algorithm: The choice between a hard or soft formulation (and the specific weight chosen) changes energy use by 7% to 33%. In contrast, at high weights, the difference between MPC and RL algorithms is a mere 2%.
- The Shared Frontier: When plotted on an energy-vs-comfort graph (Figure 4), all three algorithms lie on nearly the same Pareto frontier. This suggests that for this HVAC problem, MPC and RL are reaching essentially the same optimal operating points; the "weight" parameter just acts as a dial to move the controller along that curve.
- Hard is just "Very Soft": The results show that hard-constrained formulations behave almost identically to soft formulations with very high penalty weights.
- The "Effective" Weight Range: The trade-off between energy and comfort is most sensitive when the weight $w$ is between 0.05 and 0.4. Outside this range, you either don't heat at all or you hit a point of diminishing returns where massive energy increases yield tiny comfort gains.
Key Results
The researchers mapped the energy-comfort trade-off across different weights, as shown in Figure 5.
Key quantitative findings include:
- MPC Performance: Hard-formulation MPC used 6.8% more energy than soft MPC (at the highest weight) but achieved a 33% lower comfort violation cost.
- RL Performance: Lagrangian RL used roughly 30% more energy than its soft-weighted counterparts but significantly improved bound satisfaction.
- Algorithm Comparison: At the highest weights, the energy use of MPC, SAC, and DDPG differed by at most 2%. As seen in the inset of Figure 4, the frontiers for all three weighted controllers lie extremely close to each other.
Limitations and Future Work
- Single Case Study: The results are based on one specific building type and climate (Brussels). The authors note that different climates or building types might affect the formulations differently.
- Perfect Forecasts: The MPC was given exact forecasts. In real-world scenarios, forecast errors might degrade MPC performance more than RL, which can be more robust to uncertainty.
- Heating Only: The simulation only covered heating. Future work could examine cooling or multi-zone systems where the trade-offs might be more complex.
Summary updated