Deep Reinforcement Learning for Heat Pump and Underfloor Heating Control: An End-to-End Case Study

Antonio Manjavacas, David Suárez-Caro, Alejandro Campoy-Nieves, Guillermo Rey, Gonzalo Satrústegui, Miguel Molina-SolanaarXiv 2026 · arXiv

This paper presents a rare "sim-to-real" success story in the HVAC world. The authors developed and deployed a Deep Reinforcement Learning (DRL) controller to manage an integrated heat pump and underfloor heating system in a real, occupied home. By training an agent on a high-fidelity digital twin, they achieved robust thermal comfort and dynamic control without the need for risky online fine-tuning in the physical building.

TL;DR

The researchers successfully bridged the gap between simulation and reality by deploying a DRL-based controller in an occupied residential building to manage underfloor heating and heat pumps. Their "TQC" agent, trained exclusively on a digital twin, maintained precise thermal comfort while navigating the complex, slow-moving thermal dynamics of a real-world home.

What Problem Does This Paper Address?

Underfloor heating (UFH) is the "slow giant" of the HVAC world. Because it relies on heating a massive concrete slab, there is a significant lag—often hours—between a control action (like opening a valve) and a change in room temperature. This "high thermal inertia" makes traditional reactive controllers, such as Proportional-Integral (PI) systems, notoriously inefficient; they often overshoot the target temperature or fail to react in time to changing weather.

While Deep Reinforcement Learning (DRL) is theoretically perfect for this kind of predictive, multi-objective optimization (balancing comfort vs. energy), most research stops at the simulation stage. This paper tackles the practical "sim-to-real" challenge: How do you train an AI in a virtual environment and trust it enough to run a real family's heating system?

Methods

The authors adopted a rigorous two-phase methodology to move from code to concrete:

  1. The Pilot Study: Using a generic simulation model (Stockholm climate), they tested various DRL algorithms—including Soft Actor-Critic (SAC), Proximal Policy Optimization (PPO), and Truncated Quantile Critics (TQC). They also compared three control "Cases," ultimately finding that "Full Control" (Case 3)—where the agent simultaneously manages the heat pump's supply water temperature and individual room valve positions—offered the best performance.
  2. The Case Study & Deployment: They built a "Digital Twin" of a real, three-story residential building in Madrid using EnergyPlus. After training the best-performing agent (TQC) on this twin for 200 simulated episodes, they deployed it directly into the physical building's control hardware.
Figure 1: Overview of the proposed two-phase methodology used to design, train, and validate the DRL controller in simulation and real-building operation

Datasets

The "data" in this study primarily consists of:

  • Simulation Environments: High-fidelity EnergyPlus models of a semi-detached house (Pilot) and a real Madrid residence (Case Study).
  • Weather Data: EnergyPlus Weather (EPW) files for Stockholm and Madrid, augmented with Ornstein–Uhlenbeck noise to ensure the agent learned to be robust against unpredictable weather fluctuations.
  • Real-world Operational Data: Collected during the deployment phase (March to April 2026) to validate the agent's performance against the digital twin's predictions.

Key Insights and Ideas

  • Off-Policy Wins: Off-policy algorithms like TQC and SAC significantly outperformed on-policy ones like PPO. They were more sample-efficient and stable, which is critical when the "cost" of a training episode (a full heating season) is high.
  • The Robustness "Aha": In the pilot study, the authors intentionally "degraded" the virtual building—reducing insulation and increasing air leaks. The DRL agent proved remarkably resilient, maintaining comfort even in these "broken" buildings, though it naturally consumed more energy to do so.
  • Solar Gain is the Great Disturber: One of the biggest discrepancies between the digital twin and reality was solar radiation. In the real house, passive solar heating through large windows often pushed temperatures above the comfort limit. Since the agent could only control heating, it had to learn to "get out of the way" by closing valves completely during sunny periods.
  • Sim-to-Real is a Sensor Game: The authors found that some variables used in simulation (like diffuse solar radiation) weren't available from the real building's sensors. They had to retrain the agent with a reduced observation space to match the physical reality.

Key Results

  • Comfort Maintenance: In the real-world deployment, the TQC agent successfully kept indoor temperatures within the desired comfort range, with only minor violations (less than 1°C) in a single zone (the Bathroom-Dressing area).
  • Operational Stability: The agent learned a stable policy, maintaining the heat pump supply temperature around a steady 34°C rather than frequently cycling the equipment on and off, which helps extend the system's lifespan.
  • Scalability: The transition from a simple 5-zone pilot building to a complex 13-zone real building was successful, proving the DRL approach can scale to larger residential structures.

Limitations and Future Work

The study highlighted several "real-world" headaches that simulations often ignore:

  • Communication Reliability: The UFH valves were controlled via Wi-Fi smart relays. Intermittent connectivity issues sometimes delayed the agent's actions, creating a mismatch between the AI's intent and the system's state.
  • Occupant Interference: Real people change setpoints manually. The authors noted that human intervention can counteract the DRL policy, making it harder to distinguish the AI's performance from human behavior.
  • Seasonal Benchmarking: The real-world test occurred during a mild transition period (late winter to early spring). Future work will focus on testing the system during extreme peak-cold conditions and extending the framework to the cooling season.

Summary updated

Abstract

Efficient control of underfloor heating (UFH) systems coupled with heat pumps (HPs) is challenging due to the high thermal inertia of buildings and the slow dynamic response of UFH systems. Deep Reinforcement Learning (DRL) offers a data-driven control approach capable of simultaneously optimizing energy consumption and thermal comfort. However, the application of DRL-based controllers in occupied buildings under real operating conditions remains limited. This work presents and end-to-end case study (development, evaluation, and real-world deployment) of a DRL-based controller for an integrated UFH-HP system. First, a simulation-based pilot study was conducted to assess different DRL algorithms, control strategies, and configurations. Subsequently, a digital twin of a real residential building was developed and used to train DRL agents under simulated operating conditions. The best-performing agent was then deployed in the physical building without additional online fine-tuning. The results show that the controller successfully maintained indoor temperatures within the prescribed comfort limits while dynamically adjusting the HP supply temperature and UFH valve positions. These findings demonstrate the feasibility of deploying DRL controllers in occupied buildings and highlight their robustness and the practical challenges associated with transferring control policies from digital environments to real heating systems.
BibTeX
@misc{manjavacas2026deep,
  author = {Antonio Manjavacas and David Suárez-Caro and Alejandro Campoy-Nieves and Guillermo Rey and Gonzalo Satrústegui and Miguel Molina-Solana},
  title = {{Deep Reinforcement Learning for Heat Pump and Underfloor Heating Control: An End-to-End Case Study}},
  year = {2026},
  eprint = {2611.23063},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2611.23063},
}
Figure