Comfort as a Constraint or as a Cost in Reinforcement Learning and Model Predictive Building Control

Joram Liebeskind, Valentin Bhend, Varsha N. Behrunani, Alisa RupenyanarXiv 2026 · arXiv

TL;DR

This paper decouples the choice of control algorithm (MPC vs. RL) from the mathematical formulation of thermal comfort (hard vs. soft constraints) in building HVAC systems. The researchers found that the way comfort is framed and weighted impacts energy performance far more significantly than the choice of the algorithm itself.

What Problem Does This Paper Address?

Building climate control is a classic multi-objective optimization problem: how do we minimize electricity use while keeping indoor temperatures within a "comfort zone"? Traditionally, researchers have approached this in two ways:

  1. Hard Constraints: The controller is strictly forbidden from violating temperature bounds.
  2. Soft Constraints: Violations are allowed but penalized in the cost function using a weight ($w$).

While many studies compare Model Predictive Control (MPC) and Reinforcement Learning (RL), they often use different comfort formulations for each. This makes it impossible to tell if a performance gap is due to the algorithm's intelligence or simply how "strictly" it was told to prioritize comfort. This paper isolates these variables to provide a fair "apples-to-apples" comparison.

Methods

The authors evaluated three primary algorithms: Model Predictive Control (MPC), Soft Actor-Critic (SAC), and Deep Deterministic Policy Gradient (DDPG).

For each algorithm, they implemented:

  • Soft Formulations: A weighted sum of energy and comfort violations. They tested a grid of twelve weights ($w$) to map out the trade-off.
  • Hard Formulations:
    • For MPC, they used an "exact penalty" method (a very high weight that mimics a hard constraint).
  • For RL, they used Lagrangian methods (SAC-Lagrangian and primal-dual DDPG), where the penalty weight is automatically adapted during training to satisfy a constraint on average.

The control loop for these systems is visualized in Figure 1, showing how the controller interacts with the heat pump and floor heating system based on measurements and forecasts.

The BOPTEST control loop

Datasets

The study utilized the BOPTEST (Building Optimization Testing) framework, specifically the bestest_hydronic_heat_pump case. This is a high-fidelity emulator of a single-zone residential building in Brussels equipped with a 15 kW air-to-water heat pump. The simulation includes a full year of realistic disturbance data, including weather and occupancy patterns for a family of five.

Key Insights and Ideas

  • Formulation Trumps Algorithm: The choice between a hard or soft formulation (and the specific weight chosen) changes energy use by 7% to 33%. In contrast, at high weights, the difference between MPC and RL algorithms is a mere 2%.
  • The Shared Frontier: When plotted on an energy-vs-comfort graph (Figure 4), all three algorithms lie on nearly the same Pareto frontier. This suggests that for this HVAC problem, MPC and RL are reaching essentially the same optimal operating points; the "weight" parameter just acts as a dial to move the controller along that curve.
  • Hard is just "Very Soft": The results show that hard-constrained formulations behave almost identically to soft formulations with very high penalty weights.
  • The "Effective" Weight Range: The trade-off between energy and comfort is most sensitive when the weight $w$ is between 0.05 and 0.4. Outside this range, you either don't heat at all or you hit a point of diminishing returns where massive energy increases yield tiny comfort gains.

Key Results

The researchers mapped the energy-comfort trade-off across different weights, as shown in Figure 5.

Energy and comfort cost vs. weight

Key quantitative findings include:

  • MPC Performance: Hard-formulation MPC used 6.8% more energy than soft MPC (at the highest weight) but achieved a 33% lower comfort violation cost.
  • RL Performance: Lagrangian RL used roughly 30% more energy than its soft-weighted counterparts but significantly improved bound satisfaction.
  • Algorithm Comparison: At the highest weights, the energy use of MPC, SAC, and DDPG differed by at most 2%. As seen in the inset of Figure 4, the frontiers for all three weighted controllers lie extremely close to each other.
Energy vs. comfort cost Pareto frontier

Limitations and Future Work

  • Single Case Study: The results are based on one specific building type and climate (Brussels). The authors note that different climates or building types might affect the formulations differently.
  • Perfect Forecasts: The MPC was given exact forecasts. In real-world scenarios, forecast errors might degrade MPC performance more than RL, which can be more robust to uncertainty.
  • Heating Only: The simulation only covered heating. Future work could examine cooling or multi-zone systems where the trade-offs might be more complex.

Summary updated

Abstract

A building controller that keeps a zone within comfort bounds at minimum energy can treat the bounds as a soft constraint, by adding the weighted violation to its cost, or as a hard constraint. We analyze the effect of the hard and the soft formulation of the same problem and isolate the effect of the formulation from the effect of the algorithm. To this end, we measure the difference in energy use between the two formulations on the BOPTEST bestest_hydronic_heat_pump case, in model predictive control (MPC) and in reinforcement learning (RL). The soft formulations are a weighted MPC and weighted SAC and DDPG learners, each at twelve weights. The hard formulations are an MPC with an exact penalty and the Lagrangian learners SAC-Lagrangian and primal-dual DDPG. We find that the hard formulations use more energy than the soft formulations at the largest weight, but violate the comfort bounds less. For a large weight, the soft formulation approaches the behavior of the hard formulation. The frontiers of the energy-comfort trade-off of the three weighted controllers lie close to each other, and the weight selects an operating point on this frontier. Under the soft formulation, the three algorithms reach very similar operating points, although at somewhat different weights. For this HVAC problem, the formulation and the weight thus affect the energy use more than the choice of algorithm.
BibTeX
@misc{liebeskind2026comfort,
  author = {Joram Liebeskind and Valentin Bhend and Varsha N. Behrunani and Alisa Rupenyan},
  title = {{Comfort as a Constraint or as a Cost in Reinforcement Learning and Model Predictive Building Control}},
  year = {2026},
  eprint = {2611.20889},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.20889},
}
Figure