Abstract
Managing energy in modern residential buildings requires balancing fluctuating electricity prices, intermittent renewable generation, and complex thermal dynamics. While Model Predictive Control (MPC) is effective at handling constraints, it relies on accurate physical models that are often unavailable. Conversely, Reinforcement Learning (RL) is model-free but suffers from poor data efficiency and safety violations. This paper proposes a hybrid approach: a parameterized MPC-based RL framework. By using an MPC solver as a differentiable policy approximator, the system achieves near-optimal performance with significantly less data. In a Nordic home simulation, the proposed CDDAC-GQ algorithm outperformed Deep RL by 37.7% in cost efficiency and trained 1,000 times faster.
Introduction
The rise of smart grids and residential Distributed Energy Resources (DERs) like PV panels and batteries has made Home Energy Management Systems (HEMS) critical. The goal is to minimize costs while maintaining occupant comfort. Traditional MPC is the standard for its ability to anticipate future states and respect constraints. However, building an accurate "digital twin" of a house is costly and difficult due to unknown thermal properties. Reinforcement Learning offers a data-driven alternative but is often impractical for real-world deployment due to the millions of interactions required for convergence and its "black-box" nature regarding safety constraints.
Methods
The authors introduce a framework where the MPC itself is the "Actor" in an Actor-Critic RL setup.
- Parameterized MPC: The physics model within the MPC is treated as a set of tunable parameters ($\theta$). This allows the controller to start with a simplified, potentially inaccurate model and refine it through experience.
- CDDAC-GQ Algorithm: A novel algorithm—Compatible Delayed Deterministic Actor-Critic with Gradient Q-learning—is employed. The Critic evaluates actions using Gradient Q-learning, while the Actor (the MPC) updates its parameters to maximize the Critic's value function.
- Differentiable Optimization: To update the MPC parameters, the authors utilize the Karush-Kuhn-Tucker (KKT) conditions. This mathematical bridge allows gradients to flow from the RL objective back into the MPC's internal optimization variables, enabling end-to-end learning.
Experimental Setup
The system was tested using a high-fidelity simulation of a Nordic smart home. The setup includes an air-to-water heat pump, a hot water tank (HWT), floor heating, PV panels, and a 5kWh battery.
To simulate real-world conditions, the controller was intentionally provided with a mismatched model (4 states) compared to the "true" environment (8 states). Real-world data from Nord Pool (electricity prices) and local weather forecasts were used to drive the simulation.
Key Results
The proposed method demonstrated significant advantages over standard Deep RL (TD3):
- Data Efficiency: The MPC-based agent reached optimal performance in just 1,000 steps, while TD3 required 1.1 million steps. This makes the system viable for real-world learning where data is expensive and time-consuming to collect.
- Cost and Comfort: The CDDAC-GQ algorithm achieved a 37.7% improvement in total cost (a composite of energy prices and comfort penalties) over TD3. It maintained indoor temperatures within the 20°C–24°C range far more consistently.
- Interpretability and Speed: Unlike neural networks, the tuned parameters in this model correspond to physical or economic weights, making the system's behavior auditable. The solver's 0.08s execution time ensures real-time feasibility.
Conclusion
By embedding physical structure into a learning framework, this research bridges the gap between classical control and modern AI. The results suggest that HEMS can be both adaptive and safe, learning to exploit thermal inertia and price volatility without requiring decades of training data. Future work will focus on real-world deployment and handling non-Gaussian uncertainties in renewable generation.
Summary updated