Abstract
Despite the promising potential of energy management technologies in optimizing electric vehicle (EV) performance and fostering global energy sustainability, the extensive research conducted over the past decade has yet to translate into practical applications. This discrepancy arises primarily from the reliance of existing methodologies on simulation-based development paradigms, leading to a significant disparity between simulated results and real-world efficacy. Herein, we present a pioneering real-world data-driven energy management strategies (EMS) approach that utilizes an innovative offline reinforcement learning (ORL) framework. This paradigm enables EMS to learn from diverse real-world data, obviating the need for explicit rule design or high-fidelity simulators, and allowing for seamless application of the proposed method to any existing EMS. Moreover, it continuously enhances performance even after deployment in actual energy management systems. We evaluate the proposed ORL method on fuel cell EVs, training the ORL agent to optimize energy consumption and system degradation. The EV monitoring and management platform in China provides real-world data for validating our methodology. The results demonstrate that ORL consistently learns superior EMS in various conditions. With increasing data availability, its performance improves significantly, from 88% to 98.6% relative to theoretical optimality after two data updates. After training with more than 60 million kilometers of data, the ORL agent can learn a general EMS that adapts to unseen and corner-case conditions. These results highlight the effectiveness of integrating the data-driven method with established EMS techniques to enhance performance and underline its potential to utilize large-scale data to improve vehicle energy efficiency and longevity.
The Challenge: Bridging the Sim-to-Real Gap
Energy Management Systems (EMS) are the "brains" of hybrid and fuel cell electric vehicles, deciding in real-time how to distribute power demands between energy sources like batteries and fuel cells. Traditionally, these systems rely on rigid rule-based designs or simulation-based reinforcement learning. However, simulators often fail to capture the stochastic nature of real-world traffic and driver behavior, creating a performance gap when lab-trained agents are deployed on actual roads.
A Real-World Data-Driven Framework
To overcome these limitations, this research introduces an Offline Reinforcement Learning (ORL) framework. Unlike standard RL, which learns through real-time interaction, ORL learns entirely from static datasets of historical driving logs. This allows for the development of high-performance EMS without the need for perfect simulators or risky online exploration.
The framework employs an extended Twin Delayed Deep Deterministic Policy Gradient (TD3) algorithm. Key features include Behavior Cloning (BC) regularization to keep the agent's actions realistic and a Discriminator Blend (DB) mechanism that filters the best strategies from even suboptimal or "noisy" data.
Leveraging Large-Scale Fleet Data
The system was validated using data from an EV monitoring platform in China, encompassing a massive dataset equivalent to over 60 million kilometers of driving. This real-world context is crucial for capturing the nuances of actual vehicle operation.
Key Results and Insights
The ORL agent demonstrated remarkable performance across several metrics:
- Near-Optimality: Efficiency improved from 88% to 98.6% of the theoretical maximum after just two data updates.
- Learning from Suboptimal Data: The agent successfully distilled expert-level strategies even when trained on entirely random, low-quality data.
- Enhanced Longevity: By optimizing the fuel cell's operating points, the system significantly reduced degradation costs.
- Generalization: Training on large-scale data allowed the agent to handle "corner cases," such as high-speed highway driving, that were not present in standard training cycles.
Conclusion
This work demonstrates that integrating data-driven methods with established EMS techniques can significantly enhance vehicle energy efficiency and longevity. By creating a "data flywheel" where fleet data continuously improves the management policy, manufacturers can bridge the sim-to-real gap and deploy truly optimized energy management systems.
Summary updated