TL;DR
This paper explores how well a pre-trained Deep Reinforcement Learning (DRL) agent can adapt to control residential HVAC systems in different homes and with varying user comfort preferences. The study demonstrates that a DRL agent trained on one building model can achieve significant cost savings (up to 30% in simulations and 21% in real-world deployment) when applied to new environments, highlighting its potential for practical, cost-effective smart home energy management.
What Problem Does This Paper Address?
The building sector is a major energy consumer, with HVAC systems accounting for a significant portion of household energy use. Optimizing HVAC control is crucial for reducing energy costs and environmental impact. While Deep Reinforcement Learning (DRL) shows promise for intelligent HVAC control, a major challenge is its adaptability. Training a DRL agent online in a real house is impractical due to long learning times and potential comfort issues. Pre-training a DRL agent on a building model is an alternative, but developing accurate models for every individual home is not cost-effective. This paper addresses the gap in understanding how well a DRL agent, pre-trained on a single building model, can adapt to control HVAC in different residential settings with varying user comfort needs and building characteristics, and what challenges arise during real-world deployment.
Methods
The authors developed an end-to-end workflow for DRL-based HVAC control. This involved:
- Building Simulation Testbed Development: Creating a grey-box model of a two-zone single-family house using coupled ordinary differential equations, representing thermal resistance and capacitance. This model was trained and validated using historical data from a research house.
- Synthetic House Generation: To test adaptability, they generated 50 synthetic house models by varying parameters of the original building model, creating diverse thermal responses.
- DRL Model Development (DQN): A Deep Q-Network (DQN) was employed for HVAC control, trained offline using the simulation testbed. The DQN uses an evaluation and a target network for stability, and an experience replay buffer for efficient learning. The action space was designed to translate ON/OFF commands into setpoint adjustments within a defined flexibility band.
- Virtual Deployment: A simulation environment was used to emulate real-world deployment, allowing evaluation of the pre-trained RL model on synthetic houses without physical deployment.
- Real-World Deployment: The pre-trained RL model was deployed in the actual research house under two scenarios: one trained on the house's data and another trained on a synthetic house model.
Datasets
- Historical Data: Collected from a two-story research house in Knoxville, TN, used for training and validating the building simulation model.
- Typical Meteorological Year (TMY) Data: Hourly weather data for Knoxville, TN, used for RL training and simulation. Linear interpolation was used to generate minute-level data.
- Time-of-Use (TOU) Electricity Price Data: A dynamic price signal with multiple peak and off-peak periods was used to incentivize cost-saving behaviors.
Key Insights and Ideas
- Adaptability is Achievable: A DRL agent pre-trained on one building model can effectively adapt to control HVAC in different house environments and with varying user comfort preferences.
- Feature Scaling is Key: Scaling indoor temperature features using the defined comfort range (flexibility band) proved crucial for the RL agent's adaptability to changing setpoints and house characteristics. This removes the dependency on absolute temperature values.
- Pre-cooling for Savings: The RL agent leverages low electricity price periods for "pre-cooling" to reduce overall energy costs, a strategy not employed by a simple rule-based baseline.
- Simulation-to-Real-World Gap: While simulations show promising savings, real-world deployment results are slightly lower. This is attributed to differences in sensor measurements (DAQ vs. thermostat) and the discrete control interval (15 min).
- Synthetic Houses as Proxies: The use of synthetic houses effectively simulates the performance of the RL agent in diverse, unseen environments, providing a valuable tool for pre-deployment validation.
Key Results
- Simulation Performance: The pre-trained RL model achieved approximately 30% cost reduction compared to a baseline fixed setpoint control in simulation environments.
- Real-World Performance: In actual house deployments, the RL model achieved cost savings ranging from 11% to 21%.
- Comfort Violation: While the RL model generally maintained comfort, some minor violations were observed, particularly when setpoints changed abruptly. The baseline also experienced comfort violations.
- Energy Consumption: In some scenarios, the RL model consumed slightly more energy than the baseline, which is attributed to pre-cooling strategies and potentially larger temperature differentials.
Limitations and Future Work
- Sensor Discrepancy: The difference between DAQ and thermostat temperature measurements impacted the model's accuracy in real-world deployment.
- Control Interval: The 15-minute control interval for setpoint changes led to HVAC cycling and potential inefficiencies.
- Future Work:
- Retrain the building model using thermostat measurements for improved accuracy.
- Reduce the control timestep (e.g., to 5 minutes) or increase the setpoint change magnitude to ensure HVAC stays on/off for the entire interval.
- Investigate online learning to further improve adaptability when the pre-trained model doesn't perfectly match the target building.
- Quantify the impact of varying flexibility ranges and price signal dynamics on RL's cost-saving capabilities.
- Explore the use of look-ahead information for changing setpoints to better manage comfort.
Summary updated