Abstract
In the evolving landscape of the Smart Grid, "prosumers"—entities that both produce and consume energy—require sophisticated strategies to manage energy transactions and storage. This paper introduces a framework based on Factored Markov Decision Processes (FMDPs) to optimize daily energy management. By decomposing the state space into interacting factors, we provide an exact optimal strategy for buying, selling, and storing energy. Experimental results demonstrate that our custom Value Iteration (VI) algorithm significantly outperforms state-of-the-art general-purpose stochastic planners like SPUDD in computational efficiency while maintaining policy optimality.
Introduction
The emergence of microgrids and residential renewable energy sources has transformed consumers into prosumers. These entities face a complex optimization problem: balancing intermittent renewable production and unpredictable household consumption with fluctuating market prices. Effective management requires deciding when to purchase energy, when to sell excess production, and how to utilize battery storage to minimize costs. Previous approaches often simplified these dynamics by ignoring storage or assuming deterministic environments. This work addresses these challenges by modeling the prosumer’s environment as a stochastic process with physical constraints.
Methodology
We model the prosumer’s decision-making process as a Factored Markov Decision Process (FMDP). Unlike standard MDPs that treat the state as a single monolithic variable, FMDPs represent the state through discrete factors:
- State Variables: Time of day, battery state-of-charge, and current utility tariffs.
- Actions: Energy transaction volumes (buy/sell), battery charging/discharging rates, and tariff selection.
- External Signals: Predicted renewable production and community consumption.
To solve the FMDP, we employ a Finite-Horizon Value Iteration (VI) algorithm. The framework integrates specialized forecasting sub-models: the RENES tool for solar and wind production, and Gaussian Processes (GPs) with a Gaussian kernel for highly accurate load prediction.
Experimental Setup
The model was validated using real-world residential load data from the Public Service Company of New Hampshire, representing a 30-household community. The simulated microgrid configuration included 60kW of photovoltaic modules, 1000kW of wind turbines, and a 24-battery storage bank.
Key Results and Discussion
A central finding of this research is the efficiency of the custom VI method compared to general-purpose solvers. We compared our approach against SPUDD, a state-of-the-art solver utilizing Algebraic Decision Diagrams (ADDs). While SPUDD is designed to exploit logical symmetries, the energy management problem lacks the specific internal structure required for ADDs to be efficient.
As shown in the performance data, the custom VI method consistently produced optimal policies in significantly less time than the total time required by SPUDD (Script + Runtime). For a 24-hour horizon, our method completed computations in 1.76 to 15.84 hours, representing roughly 15% to 25% of the time taken by SPUDD. Importantly, the policies generated were identical in quality to the optimal benchmarks, proving that the speedup did not come at the cost of decision quality.
Conclusion and Future Work
This work demonstrates that factoring the state space allows for compact representation and efficient optimization of prosumer behavior. Future research will address the limitations of discretization by exploring continuous state space models, such as Fitted Value Iteration. Additionally, we aim to extend this framework to coordinate behavior within renewable energy source cooperatives, enabling multiple prosumers to collaborate for greater collective benefit.
Summary updated