Safe Reinforcement Learning-based Control for Hydrogen Diesel Dual-Fuel Engines

Vasu Sharma, Alexander Winkler, Armin Norouzi, Jakob Andert, David C. Gordon, Hongsheng GuoarXiv 2025 · DOI · arXiv

Abstract

Hydrogen-Diesel Dual-Fuel (H2DF) engines represent a critical transitional technology for decarbonizing heavy-duty transport. However, their complex, non-linear combustion dynamics pose significant control challenges, particularly regarding safety and emission constraints. This paper presents a data-driven control framework using safe Reinforcement Learning (RL). By training RL agents on a Gated Recurrent Unit (GRU)-based "digital twin" of a modified Cummins engine, we develop a controller that respects strict physical and environmental limits. The resulting system demonstrates high-fidelity tracking, operates 6x faster than traditional Model Predictive Control (MPC) on low-cost hardware, and effectively balances engine load with emission standards.

Introduction

Decarbonizing heavy-duty transport is a massive challenge. While hydrogen is a promising clean fuel, completely replacing diesel infrastructure overnight is impractical. Hydrogen-Diesel Dual-Fuel (H2DF) engines offer a bridge, but they are a control nightmare. Adding hydrogen to a diesel cycle introduces complex, non-linear combustion characteristics that traditional mathematical models often fail to capture accurately. While Reinforcement Learning (RL) can learn these patterns, it typically requires millions of samples, making direct training on physical engines unfeasible. Furthermore, any controller must respect strict safety constraints, such as maximum pressure rise rates and NOx/soot emission standards, in real-time.

Methodology

The proposed framework utilizes a four-step methodology to bridge the gap between simulation and real-world application: system identification, digital twin development, offline RL training, and online evaluation.

Overview of the four-step methodology for safe RL-based engine control.

A key innovation is State Augmentation, where the internal hidden states of the GRU model are fed back into the RL agent. This provides temporal memory, significantly improving learning efficiency. Safety is maintained via a "penalty polytope" approach in the reward function, which imposes exponential penalties for predicted constraint violations.

Datasets and Experimental Setup

The study utilized a dataset of 85,000 consecutive engine cycles from a modified Cummins QSB 4.5L 4-cylinder engine. The data was split into 80% for training, 15% for validation, and 5% for testing. The model tracks four control inputs (Diesel/H2 injection duration and timing) and four outputs (Engine load/IMEP, NOx, Soot, and Maximum Pressure Rise Rate/MPRR).

Key Insights

  • Hidden State Utility: State augmentation allowed agents to reach higher rewards faster, suggesting the agents leverage the model's internal representation of engine dynamics.
  • Algorithmic Strategies: PPO and TD3 agents converged on different strategies; PPO favored hydrogen for load demands, while TD3 preferred diesel to manage NOx constraints.
  • Soft-Safety Effectiveness: L1-style penalties in the reward function successfully taught agents to respect physical limits without hard-coded safety layers.
  • Hardware Efficiency: The control policy was successfully deployed on a Raspberry Pi 400, demonstrating feasibility for edge computing in industrial settings.

Results and Discussion

The GRU-based engine model achieved high fidelity with a Root Mean Squared Percentage Error (RMSPE) below 4.6%. In terms of performance, the RL controller achieved a 1 ms turnaround time—6 times faster than a GRU-based MPC optimization.

Detailed view of the online reinforcement learning controller evaluation setup highlighting state augmentation.

Validation tests showed excellent generalization over long trajectories with random load changes, maintaining all emission and pressure constraints. The agents demonstrated robust tracking even in scenarios eight times longer than the training trajectories.

Conclusion and Future Work

This work demonstrates a robust framework for applying safe RL to complex engine control. Future research will focus on online fine-tuning to adapt to engine wear and environmental changes, as well as extending the framework to 100% hydrogen combustion engines.

Summary updated

BibTeX
@misc{sharma2025safe,
  author = {Vasu Sharma and Alexander Winkler and Armin Norouzi and Jakob Andert and David C. Gordon and Hongsheng Guo},
  title = {{Safe Reinforcement Learning-based Control for Hydrogen Diesel Dual-Fuel Engines}},
  year = {2025},
  eprint = {2502.09826},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.09826},
}
Figure