Introduction
Managing energy in modern urban grids is a complex balancing act. Operators must minimize peak demands, reduce carbon emissions, and lower ramping costs while strictly adhering to physical constraints such as energy balancing. Traditional Reinforcement Learning (RL) approaches often struggle in this domain due to long training periods, a lack of interpretability, and difficulties in strictly satisfying safety constraints. Conversely, traditional optimization methods like Model Predictive Control (MPC) are robust for constraint handling but often lack the flexibility to adapt to changing real-world conditions like weather or occupant behavior.
This paper introduces ZO-iRL (Zeroth-Order Implicit Reinforcement Learning), the winning solution for the 2021 CityLearn Challenge. ZO-iRL bridges the gap between optimization and learning by using the solution of a convex optimization problem as a control policy and tuning its parameters via a novel "guided" evolutionary search.
Methodology: ZO-iRL Framework
The core of ZO-iRL is treating the "solution function" of an optimization problem as the agent's policy.
Optimization as Policy
Instead of a neural network, the agent solves a convex optimization problem at every time step to decide its actions, such as charging or discharging batteries. This allows the agent to explicitly encode physical constraints and plan for future states. The optimization model utilizes "virtual electricity prices" ($\zeta$) as adaptive parameters. By adjusting these prices, the agent's behavior is tuned.
Guided Evolutionary Search (ES)
To find optimal parameters without a differentiable environment model, the authors employ an Evolutionary Strategy. They sample candidate parameters, test them in the environment, and update the parameter distribution based on the resulting rewards.
Trajectory-Based Guidance
A key innovation is the introduction of a guidance signal ($\varrho$) derived from the agent's actual trajectory. While standard ES is a blind search, this guidance signal nudges the search in a specific direction based on observed performance. For instance, if high peak demand is observed at a specific time, the signal steers the evolutionary search to increase the "virtual price" for that period, effectively teaching the agent to shift its load.
Experimental Setup
The framework was evaluated using the 2021 CityLearn Challenge environment, which simulates a community of buildings with distributed energy storage. The dataset includes four years of data across five climate zones, covering weather, solar generation, and building energy demands. Performance was measured against a Rule-Based Controller (RBC) across six metrics: ramping costs, 1-load factor, average daily peak demand, maximum peak demand, total electricity consumption, and carbon emissions.
Key Results
The ZO-iRL agent secured 1st place in the 2021 CityLearn Challenge, significantly outperforming popular RL baselines such as SAC, PPO, and TD3.
- Overall Performance: Achieved a total score of 0.944 (lower is better), compared to 1.070 for the runner-up.
- Ramping Costs: Showed a substantial improvement with a score of 0.781, whereas the best baseline (MARLISA) scored 1.022.
- Peak Demand: Consistently reduced peak demand across all climate zones, achieving a score of 0.962.
- Sample Efficiency: ZO-iRL demonstrated remarkable sample efficiency, converging to a high-performing policy within the first few months of simulation, whereas standard RL baselines struggled to improve significantly even after a year.
Discussion and Conclusion
ZO-iRL offers the "best of both worlds" by nesting optimization within a learning loop, providing the mathematical rigor of classical control with the flexibility of RL. A significant advantage is interpretability: because the learned parameters are "virtual prices," operators can understand the logic behind the agent's decisions. Furthermore, the trajectory-based guidance significantly accelerates learning, which is vital for online deployment.
While highly effective, the method requires domain expertise to design the guidance signal logic. Additionally, solving an optimization problem at each step introduces computational overhead, though it remained within the challenge's limits. Future work may explore incorporating first-order methods or Bayesian optimization to further refine parameter updates.
Summary updated