Optimizing the Battery-Swapping Problem in Urban E-Bike Systems with Reinforcement Learning

Wenjing Li, Zhao Li, Xuanwu Liu, Ruihao Zhu, Zhenzhe Zheng, Fan WuIJCAI 2025 · DOI · OpenReview

TL;DR

This paper tackles the complex challenge of optimizing battery swapping for e-bikes (e-bikes) in urban delivery systems, where supply and demand imbalances at swapping stations lead to courier delays and reduced productivity. They propose a novel Wasserstein-enhanced Proximal Policy Optimization (W-PPO) algorithm, which uses Wasserstein distance to improve reinforcement learning's robustness in dynamic environments, leading to significant improvements in battery utilization and service capacity.

What Problem Does This Paper Address?

Urban e-bike delivery systems are booming, with millions of couriers relying on e-bikes for their livelihoods. A major bottleneck is the limited battery capacity of these e-bikes, leading to "range anxiety" and downtime. Battery-swapping stations offer a solution by allowing couriers to quickly exchange depleted batteries for fully charged ones. However, managing the supply of charged batteries and the demand from couriers at these stations is a complex problem. Imbalances can leave couriers stranded or force them to travel long distances to find a charged battery, directly impacting their earnings and the efficiency of delivery platforms. This paper aims to optimize this battery-swapping process to maximize courier productivity and overall system efficiency.

Methods

The core of this paper's contribution lies in its novel approach to the Battery-Swapping Problem (BSP). They first formulate the BSP as a discrete-time Markov Decision Process (MDP), which allows them to model the sequential decision-making involved in managing battery inventory.

The key innovation is the development of a Wasserstein-enhanced Proximal Policy Optimization (W-PPO) algorithm. Traditional Proximal Policy Optimization (PPO) uses Kullback-Leibler (KL) divergence to constrain policy updates, ensuring stability but potentially overlooking the true differences in policy behavior. W-PPO addresses this by incorporating the Wasserstein distance, a metric from optimal transport theory, into the policy optimization process. This allows the algorithm to more accurately measure the difference between the current and updated policies, leading to more robust learning in dynamic and uncertain environments like the e-bike battery-swapping network. The algorithm is designed to maximize the Shared Battery Utilization Ratio (SBUR), a metric representing the proportion of riders who successfully get a fully charged battery.

Datasets

The authors utilized a large-scale, real-world dataset provided by an anonymous company. This dataset includes comprehensive information on over 500,000 batteries and 30,000 battery stations across more than 80 cities. It encompasses time-location data and battery status, detailing operational metrics like inventory and capacity. The experiments were conducted using one month (31 days) of this data.

Key Insights and Ideas

  • Wasserstein Distance for Robustness: The paper highlights that simply minimizing the divergence between policies (as in standard PPO) isn't enough for complex real-world problems like BSP. The Wasserstein distance provides a more nuanced measure of policy difference, leading to better adaptation to dynamic supply-demand fluctuations.
  • Optimal Route Size Matters: The research found that performance (measured by SBUR) is not monotonically increasing with the number of stations per route. There's an optimal sweet spot (around 7 stations per route in their experiments) where the system balances coverage with efficient resource management. Too few stations limit exploitation of resources, while too many can lead to inefficiencies due to larger geographical coverage.
  • Action Space Sensitivity: The choice of action space (how many batteries a vehicle can pick up or drop off) significantly impacts performance. An overly aggressive action space can deplete station reserves, leading to premature experiment termination.
  • On-Policy vs. Off-Policy: The study observed that their proposed on-policy W-PPO method outperformed the off-policy Soft Actor-Critic (SAC) in this specific BSP context. This suggests that for problems with strong temporal dependencies and the need for precise policy updates, on-policy methods, especially when enhanced, can be more effective.

Key Results

The W-PPO algorithm demonstrated significant improvements over baseline models:

  • SBUR Improvement: A 30.59% average improvement in the Shared Battery Utilization Ratio (SBUR) was achieved for routes with 7 stations.
  • BSR Increase: A 16.09% increase in the Battery Supply Ratio (BSR) was observed, indicating improved battery availability.
  • Rider Growth: This increase in BSR translates to a 45.55% potential growth in the number of e-bike riders that can be served.
  • Battery Savings: The optimized system could save approximately 3,972 batteries.

Limitations and Future Work

The authors acknowledge several limitations:

  • Data Protection: Restrictions on data protection limited the evaluation of cost efficiency.
  • Generalizability: The method's performance is sensitive to hyperparameters, and its direct applicability to other transportation systems like EVs or AGVs might require adaptation due to differing operational dynamics.
  • Real-time Prediction: The current approach relies on forecasting SnD, and future work could focus on adapting W-PPO for real-time SnD predictions.

Future work directions include developing advanced hyperparameter optimization techniques, adapting W-PPO for broader transportation contexts, and extending it to real-time demand prediction.

Summary updated

Abstract

E-bikes (EBs) are a key transportation mode in urban area, especially for couriers of delivery platforms, but underdeveloped EB systems can hinder courier's productivity due to limited battery capacity. Battery-swapping stations address this issue by enabling riders to exchange depleted batteries for fully charged ones. However, managing supply and demand (SnD) imbalances at these stations has become increasingly complex. To address this, we introduce a new approach that formulates the Battery-Swapping Problem (BSP) as a discrete-time Markov Decision Process (MDP) to capture the dynamics of SnD imbalances. Building on it, we propose a Wasserstein-enhanced Proximal Policy Optimization (W-PPO) algorithm, which integrates Wasserstein distance with reinforcement learning to improve the robustness against uncertainty in forecasting SnD. W-PPO provides a BSP-specific, accurate loss function that reflects reward variations between two policies under real-world simulation. The algorithm’s effectiveness is assessed using key metrics: Shared Battery Utilization Ratio (SBUR) and Battery Supply Ratio (BSR). Simulations on real-world datasets show that W-PPO achieves a 30.59% improvement in SBUR and a 16.09% increase in BSR ensures practical applicability. By optimizing battery utilization and improving EB delivery systems, this work highlights the potential of AI for creating efficient and sustainable urban transportation solutions.
BibTeX
@inproceedings{li2025optimizing,
  author = {Wenjing Li and Zhao Li and Xuanwu Liu and Ruihao Zhu and Zhenzhe Zheng and Fan Wu},
  title = {{Optimizing the Battery-Swapping Problem in Urban E-Bike Systems with Reinforcement Learning}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1029},
}
Figure