RL-HEMS: Reinforcement Learning Based Home Energy Management System for HVAC Energy Optimization

Olivera KotevskaOSTI 2023 · OpenAlex

Abstract

Heating, ventilation, and air conditioning (HVAC) is one of the major energy consumers in the residential sector. It is important to be able to monitor and control the energy consumed to provide utility services such as load shaping while satisfying the comfort and economic constraints of the homeowner. The objective of this work is to create the optimal schedule for HVAC operation to reduce the cost while satisfying the home-owner and equipment’s constraints using a model-free Reinforcement Learning (RL)-based optimization. The specific goal is to find the right balance between reducing energy cost, consumption, and customer comfort level. Our research effort addresses this optimization problem using multiple components: the development of initial learning test-bed and implementation of RL techniques on a real home. This will enable the rapid evaluation of the RL techniques and provide an early baseline to train before implementation on site. The RL algorithm is designed to learn the energy use patterns and generate the optimized schedule for HVAC within an acceptable time-interval to satisfy the homeowner’s comfort and minimize the energy usage. Our preliminary results are promising and we have observed a 17% reduction in the total cost and a 15% reduction in the power utilization using our RL-based HVAC model—RL-HEMS.

Introduction

Heating, ventilation, and air conditioning (HVAC) accounts for approximately 51% of the energy consumed in the residential sector. Traditionally, homeowners manage this through simple rule-based thermostats. However, as the "smart grid" evolves, electricity prices are becoming increasingly dynamic. Static rules cannot adapt to fluctuating prices or complex thermal lag in buildings. While standard Machine Learning (ML) can predict usage, it doesn't inherently know how to act to minimize costs. The challenge is finding the "sweet spot" where energy costs are minimized without letting the house get uncomfortably hot or cold, all while accounting for the specific thermal characteristics of a unique home.

Methods

Researchers at Oak Ridge National Laboratory have developed RL-HEMS, a reinforcement learning-based system that optimizes residential HVAC schedules. By using a "look-ahead" strategy to anticipate electricity price changes and weather patterns, the system achieves significant savings while maintaining homeowner comfort.

The RL Framework

The authors propose a model-free Reinforcement Learning (RL) approach, specifically utilizing Deep Q-Learning (DQN). The system architecture involves an agent processing environmental parameters to send optimized actions to the HVAC system.

RL-HEMS system architecture overview showing the interaction between the agent, thermostat, and home environment.
  • State Space: Includes time of day, indoor temperature ($T_{in}$), outdoor temperature ($T_{out}$), and price look-ahead information.
  • Action Space: A simple binary choice: AC 'on' or 'off'.
  • Reward Function: A dual-term function that penalizes both high electricity costs and "comfort violations" (when the temperature drifts outside the user's preferred range).
  • Architecture: To stabilize learning, they used a two-network structure (an evaluation network and a target network) combined with experience replay, which allows the agent to learn from a "memory" of past states and actions.

The Simulator

Before deploying on a real home, the team developed a 5R4C "gray-box" model. This is a simplified physics-based model representing the house as an electrical circuit of five resistors (heat transfer resistance) and four capacitors (thermal capacitance). This allowed for rapid, high-fidelity training without the risks of experimenting directly on a live HVAC system.

The 5R4C thermal model representing the home's heat transfer characteristics through an equivalent electrical circuit.

Datasets

The model was trained and validated using high-resolution data from a real-world testbed: a highly instrumented, unoccupied 2,400 sq. ft. research home in Knoxville, Tennessee. Data was collected at one-minute intervals, including indoor/outdoor temperatures, HVAC power consumption, and sensible cooling capacity. The study specifically focused on summer data (July 2018) to test cooling optimization during peak heat.

Key Insights and Results

The Power of Look-Ahead

Unlike previous RL attempts that only looked at instantaneous prices, RL-HEMS uses future price signals. This allows the agent to "pre-cool" the home when electricity is cheap, using the building's thermal mass as a battery to stay cool when prices spike.

Performance Metrics

The RL-HEMS was compared against a standard rule-based baseline model, yielding the following results:

  • Cost Savings: Achieved a 17% reduction in total monthly electricity costs ($83.39 vs $101.17).
  • Energy Efficiency: Reduced total power utilization by 15% (366.7 kWh vs 433.23 kWh).
  • Comfort Maintenance: The system successfully kept the indoor temperature within the user-defined comfort zone ([19°C, 24°C]) for the vast majority of the time.

Conclusion

While the results are impressive, future work will expand the model to include heating and ventilation, investigate the impact of different action intervals (the "k" factor), and address the complexities of multi-zone buildings and varying occupancy. Preliminary findings suggest that RL-HEMS provides a robust baseline for intelligent, cost-aware home energy management.

Summary updated

Abstract

Heating, ventilation, and air conditioning (HVAC) is one of the major energy consumers in the residential sector. It is important to be able to monitor and control the energy consumed to provide utility services such as load shaping while satisfying the comfort and economic constraints of the homeowner. The objective of this work is to create the optimal schedule for HVAC operation to reduce the cost while satisfying the home-owner and equipment’s constraints using a model-free Reinforcement Learning (RL)-based optimization. The specific goal is to find the right balance between reducing energy cost, consumption, and customer comfort level. Our research effort addresses this optimization problem using multiple components: the development of initial learning test-bed and implementation of RL techniques on a real home. This will enable the rapid evaluation of the RL techniques and provide an early baseline to train before implementation on site. The RL algorithm is designed to learn the energy use patterns and generate the optimized schedule for HVAC within an acceptable time-interval to satisfy the homeowner’s comfort and minimize the energy usage. Our preliminary results are promising and we have observed a 17% reduction in the total cost and a 15% reduction in the power utilization using our RL-based HVAC model—RL-HEMS.
BibTeX
@inproceedings{kotevska2023rl,
  author = {Olivera Kotevska},
  title = {{RL-HEMS: Reinforcement Learning Based Home Energy Management System for HVAC Energy Optimization}},
  booktitle = {OSTI},
  year = {2023},
}
Figure