Achieving Scalable Model-Free Demand Response in Charging an Electric Vehicle Fleet with Reinforcement Learning

Nasrin Sadeghianpourhamami, Johannes Deleu, Chris Develdere-Energy 2018 · DOI

TL;DR

This paper tackles the challenge of coordinating electric vehicle (EV) charging for demand response (DR) in a scalable way. They propose a novel Markov Decision Process (MDP) formulation that compactly represents EV fleet states and actions, allowing a batch reinforcement learning algorithm (Fitted Q-Iteration) to learn optimal charging policies without needing a pre-existing model of the system dynamics.

What Problem Does This Paper Address?

Coordinating the charging of electric vehicles (EVs) to participate in demand response (DR) programs is crucial for grid stability and efficiency. DR aims to shift electricity consumption away from peak hours. However, applying reinforcement learning (RL) to manage a fleet of EVs presents a significant challenge: the "curse of dimensionality." As the number of EVs and their individual characteristics (like arrival times, departure times, and charging needs) increase, the state and action spaces of the RL problem explode, making it computationally intractable. Existing RL solutions often focus on individual EVs or rely on heuristics for fleet-level management, which don't guarantee optimal outcomes. This paper addresses the need for a scalable, model-free approach that can handle fleets of EVs with varying characteristics.

Methods

The core of this paper is a novel Markov Decision Process (MDP) definition designed for EV fleet charging. Key innovations include:

  • Compact State Representation: Instead of tracking each EV individually, the authors propose a 2D grid to represent the fleet's aggregate state. This grid bins EVs based on their remaining departure time and their charging duration. This binning significantly reduces the state space size, making it independent of the fleet size and thus scalable.
  • Compact Action Representation: The action space is also simplified. At each time step, the decision is binary for each bin in the state grid: either charge the EVs in that bin or don't.
  • Batch Reinforcement Learning (Fitted Q-Iteration - FQI): To learn the optimal charging policy without knowing the underlying system dynamics (i.e., how the state transitions), the authors employ a batch RL algorithm called Fitted Q-Iteration. FQI uses a dataset of past experiences (state, action, next state, cost) to iteratively approximate the optimal state-action value function.

Datasets

Not discussed in the paper. The paper mentions using a "dataset of past experiences" for FQI and proposes evaluating on "real-world EV charging dataset (e.g., [3])" in future work.

Key Insights and Ideas

  • Scalability through Aggregation: The most significant insight is that by cleverly aggregating EV characteristics into a compact state representation (the 2D grid), the scalability problem of RL for EV fleets can be overcome. This allows the system to handle larger fleets without an exponential increase in computational complexity.
  • Model-Free Flexibility: The use of RL, specifically FQI, means the system doesn't need a perfect model of grid conditions or EV behavior. This is crucial in real-world scenarios where such models are often unavailable or inaccurate.
  • Balancing Charging Completion and Load Flattening: The cost function is designed to penalize both unfinished charging by departure time and excessive peak load, demonstrating a practical approach to balancing user needs with grid objectives.

Key Results

The paper presents an exemplary scenario with two EVs over a 3-hour horizon. The results show that the optimized charging policy, learned using their proposed method, successfully flattens the load curve compared to a "business as usual" (BAU) policy. Specifically, the optimized policy shifts charging to avoid simultaneous charging at the first time slot, demonstrating its ability to manage demand.

Limitations and Future Work

The authors acknowledge several limitations and outline promising future directions:

  • Larger State/Action Spaces: For longer connection times and larger fleets, the state matrix can become very large, challenging exploration in FQI.
  • Computational Demands: Longer control horizons increase FQI iterations, and larger state/action spaces necessitate more sophisticated function approximators (e.g., neural networks).
  • Exploration Strategy: For massive state spaces, sampling the environment using a tree exploration strategy will be necessary.
  • Generalization: Extending the methodology to DR use cases beyond simple load flattening is a key future goal.
  • Real-World Validation: Evaluating the proposed methodology on actual real-world EV charging datasets is planned.

Summary updated

BibTeX
@inproceedings{sadeghianpourhamami2018achieving,
  author = {Nasrin Sadeghianpourhamami and Johannes Deleu and Chris Develder},
  title = {{Achieving Scalable Model-Free Demand Response in Charging an Electric Vehicle Fleet with Reinforcement Learning}},
  booktitle = {e-Energy},
  year = {2018},
  doi = {10.1145/3208903.3212042},
}
Figure