Optimize Battery Control: A Multi-Objective Evolutionary Ensemble Reinforcement Learning Approach

Jingwei Hu, Kai Xie, Zheng Fang, Xiaodong Li, Junchi Yan, Zhihong ZhangIJCAI 2025 · DOI

TL;DR

This paper introduces an Evolutionary Ensemble Reinforcement Learning (EERL) framework to optimize control strategies for Dynamically Reconfigurable Battery (DRB) systems. EERL addresses the challenge of multi-objective optimization in DRB systems, leading to a significant 11.08% increase in energy efficiency and improved safety compared to existing methods.

What Problem Does This Paper Address?

Lithium-ion battery energy storage systems are crucial for various applications, but their performance is heavily influenced by the consistency and aging of individual cells. Traditional battery management systems struggle with multi-objective optimization, often leading to imbalanced performance and inefficient energy utilization. Dynamically Reconfigurable Battery (DRB) systems, which use high-speed power electronic switches to adjust interconnections in real-time, offer a solution. However, controlling these systems presents a complex multi-objective optimization problem, where balancing performance (e.g., energy efficiency) with safety (e.g., preventing short circuits) is challenging. Existing reinforcement learning (RL) approaches often fail to handle the high dimensionality of DRB configurations or get stuck in local optima when scalarizing multiple objectives.

Methods

The paper proposes an Evolutionary Ensemble Reinforcement Learning (EERL) framework. Key aspects include:

  • Ensemble Learning: The complex multi-objective task is decomposed into simpler sub-tasks, each handled by an independent agent. These agents are trained using ensemble learning, leveraging multiple reward functions corresponding to specific objectives.
  • Evolutionary Algorithm: An evolutionary algorithm is integrated to manage the population of agents. It uses non-dominated sorting to identify the best-performing agents and refines them through crossover and mutation. This approach prevents agents from getting stuck in local optima by considering Pareto optimality.
  • Buffering Mechanism: A buffering mechanism is employed to avoid direct replacement of the "actor" (the agent controlling the system). This enhances agent robustness during training by ensuring that only well-updated individuals replace existing ones.
  • Soft Actor-Critic (SAC): The RL policy within the framework uses the Soft Actor-Critic (SAC) algorithm, chosen for its ability to integrate with variational autoencoders (VAEs) for efficient selection in large-scale discrete action spaces.
  • VAE for Action Space Reduction: A Variational Autoencoder (VAE) is used to transform the high-dimensional discrete action space (representing switch configurations) into a lower-dimensional continuous latent space, making it manageable for the RL agent.

Datasets

The paper does not explicitly mention specific public datasets. Instead, it describes experiments conducted using:

  • Simulations: The DRB system is modeled and simulated using SIMULINK. The simulation uses a predefined discharge profile for eight cells and twenty-nine switches.
  • Real-World Prototype: A real DRB system is constructed and tested using INR 18650-20R model batteries.

Key Insights and Ideas

  • Decomposition is Key: Breaking down the complex multi-objective control problem into smaller, manageable sub-tasks is crucial for effective RL in DRB systems.
  • Evolutionary Optimization for Pareto Fronts: Integrating evolutionary algorithms with RL provides a robust way to navigate the Pareto front of multiple objectives, avoiding local optima that plague simpler scalarization methods.
  • Ensemble Learning Enhances Robustness: Training multiple agents on sub-tasks and combining their outputs leads to more balanced performance across objectives and improved overall system efficiency.
  • Safety is Paramount: The EERL framework explicitly incorporates safety constraints, reducing illegal connections to 0%, which is a critical improvement over other methods that may still exhibit occasional safety violations.
  • VAE for Scalability: Using a VAE to manage the vast action space of DRB configurations is essential for making RL practical in this domain.

Key Results

  • Energy Efficiency: The EERL framework achieved an 11.08% increase in energy efficiency in real-world testing compared to existing approaches.
  • SOC Variance Reduction: SOC variance was reduced from 12% to 0.427% in simulations.
  • Energy Release: Energy released per cell improved from 4.356 Wh to 4.701 Wh (a 7.920% improvement).
  • Safe Parallelism: The framework achieved 88.470% safe parallel demand and reduced illegal connections from 0.374% to 0%.
  • Reduced Switching Operations: The average number of switching actions per decision was reduced by 14.581%.
  • Efficient Inference: Real-world inference time was approximately 1.00 ms, demonstrating high efficiency.

Limitations and Future Work

  • Limitations:
    • The paper notes that while the proposed framework significantly reduces illegal connections, the complexity of hidden variables can still lead to actions that violate connection constraints. The pre-screening feature helps mitigate this.
  • The comparison with other multi-objective RL algorithms highlights that some methods struggle with convergence or achieve suboptimal results due to their specific sampling strategies or inability to handle complex reward conditions.
  • Future Work:
    • The authors intend to apply the EERL model to high-capacity energy storage systems.
  • Further analysis and optimization of DRB systems are planned.

Summary updated

BibTeX
@inproceedings{hu2025optimize,
  author = {Jingwei Hu and Kai Xie and Zheng Fang and Xiaodong Li and Junchi Yan and Zhihong Zhang},
  title = {{Optimize Battery Control: A Multi-Objective Evolutionary Ensemble Reinforcement Learning Approach}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/1024},
}
Figure