Abstract
This paper introduces RLbEMS, a Batch Reinforcement Learning system designed to autonomously manage energy storage and sales in smart homes. As homeowners transition into "prosumers" with local generation and storage capabilities, the complexity of optimizing financial returns amidst volatile pricing and weather-dependent generation increases. RLbEMS leverages historical data and price trend indexes to maximize profit and reduce peak grid demand. Evaluation on real-world datasets from Brazil and the USA shows up to a 20% increase in financial gains and a ~30% reduction in peak demand without imposing rigid appliance schedules on users.
Introduction
The evolution of smart grids has transformed homeowners into "prosumers" who both consume and generate electricity. The core challenge for an Energy Management System (EMS) is determining the optimal timing for storing surplus energy versus selling it back to the grid. Traditional rule-based systems often require intrusive appliance scheduling, which limits user adoption. This research proposes a flexible, automated approach that respects user autonomy while maximizing economic efficiency.
Methodology
The energy management problem is modeled as a Markov Decision Process (MDP) and solved using Batch Reinforcement Learning (BRL), specifically the Fitted Q-iteration (FQI) algorithm. Unlike online RL, BRL utilizes fixed historical datasets, ensuring stability and safety in physical power systems.
System Components
The state space includes battery levels, solar generation, home demand, and price trend indexes. These indexes are critical as they allow the agent to anticipate market movements by observing the velocity of price changes. The reward function incentivizes selling energy when prices exceed historical seasonal averages. A Radial Basis Neural Network is employed for function approximation to estimate Q-values.
Data Contexts
The system was validated using two distinct datasets:
- Brazil (São Paulo): Time-of-Use (TOU) tariff data combined with real solar generation and simulated demand based on statistical appliance usage.
- USA (PJM Market): Five years of Real-Time Pricing (LMP) data and simulated solar generation based on Kyocera module specifications.
Key Insights and Results
A primary innovation is the inclusion of price trends in the state vector, enabling the agent to "hold" energy during price hikes to sell at the peak. Furthermore, by focusing solely on surplus management rather than appliance scheduling, the system maintains user freedom.
Performance Metrics
RLbEMS was compared against a Naive-greedy policy and a Simple Storage Policy (SSP):
- Financial Gains: The system achieved a 20.78% profit increase in the Brazilian study and a 14.51% annual increase in the USA study.
- Peak Demand Reduction: An emergent benefit was the reduction of peak grid demand by 39.78% (Brazil) and 29.38% (USA), as high prices typically correlate with high grid stress.
Discussion and Limitations
While highly effective, the system faced challenges in scenarios with low price volatility, such as the Spring season in the USA study, where it performed slightly worse than a naive policy due to battery efficiency losses. Future work aims to incorporate Transfer Learning to allow off-line policies to adapt to real-time shifts in user behavior or market conditions. Additionally, the dependence on high-quality historical data suggests that performance may vary based on regional data availability.
Summary updated