Energy Scheduling of Hydrogen Hybrid UAV Based on Model Predictive Control and Deep Deterministic Policy Gradient Algorithm

Haitao Li, Chenyu Wang, Shufu Yuan, Hui Zhu, Bo Li, Yuexin Liu, Li SunAlgorithms 2025 · DOI

TL;DR

This paper tackles the critical challenge of energy management in hydrogen-powered hybrid UAVs by proposing a novel framework that combines Model Predictive Control (MPC) for power optimization with Deep Deterministic Policy Gradient (DDPG) for efficient energy scheduling. This hybrid approach aims to enhance UAV endurance and operational efficiency in complex and uncertain environments.

What Problem Does This Paper Address?

Unmanned Aerial Vehicles (UAVs) are increasingly deployed across various sectors, but their operational range and payload capacity are often limited by the energy density of traditional lithium-ion batteries. Hydrogen fuel cells offer a promising alternative due to their higher energy density, enabling longer endurance. However, managing the energy flow between a fuel cell, battery, and hydrogen storage in a hybrid system, especially under dynamic and unpredictable flight conditions, presents a significant challenge. Existing energy scheduling methods, such as rule-based or purely optimization-based approaches, often lack the adaptability to handle these uncertainties, leading to suboptimal performance and reduced efficiency.

Methods

The paper proposes a two-pronged approach:

  • Model Predictive Control (MPC): This optimization-based technique is used to predict and optimize the electrical power consumption of the UAV. It operates by establishing a system model, predicting future states, and then optimizing control inputs to minimize power usage while adhering to flight mission requirements. MPC is applied to each distinct flight phase (takeoff, climb, cruise, descent, landing) with tailored configurations.
  • Deep Deterministic Policy Gradient (DDPG): This deep reinforcement learning algorithm is employed for efficient energy scheduling. DDPG is chosen for its ability to handle continuous action spaces, making it suitable for optimizing the power flow between the hydrogen fuel cell and the battery. It learns an optimal policy by interacting with the environment, aiming to minimize operational costs and ensure system reliability. The DDPG algorithm is trained using a combination of state space (including flight parameters and battery SOC) and action space (hydrogen fuel cell power output).

The integration of MPC and DDPG allows for a synergistic approach: MPC optimizes the predicted power demand, and DDPG then schedules the energy sources to meet this demand efficiently, considering system constraints and operational costs.

Datasets

Not discussed in the paper. The authors mention using experimental data for system development and generating typical day training scenarios for DDPG, but specific datasets are not detailed.

Key Insights and Ideas

  • Synergy of MPC and DDPG: The core insight is that combining MPC for power prediction/optimization with DDPG for intelligent scheduling effectively addresses both the need for efficient power usage and the challenge of dynamic energy management in uncertain environments. MPC handles the predictable aspects of power demand, while DDPG provides the adaptive learning needed for real-time adjustments.
  • Addressing Uncertainty: The paper highlights how traditional methods struggle with the inherent uncertainties in UAV flight (e.g., weather, payload variations). The DDPG's reinforcement learning approach allows it to learn and adapt to these uncertainties, improving robustness.
  • Continuous Control: DDPG's suitability for continuous action spaces is crucial for fine-grained control over the hydrogen fuel cell's power output, enabling more precise energy management compared to discrete control methods.
  • Real-time Capability: The authors emphasize that DDPG offers faster online learning and decision-making compared to many traditional optimization methods, which is vital for real-time UAV operations.

Key Results

The paper demonstrates that the proposed MPC and DDPG combined approach leads to:

  • Optimized Electrical Power: The MPC method successfully predicts and optimizes electrical power consumption, resulting in values generally lower than a reference value.
  • Efficient Energy Scheduling: The DDPG algorithm achieves convergence in cumulative rewards across different flight periods, indicating effective energy scheduling.
  • Reasonable Fuel Cell and SOC Scheduling: Figures 10-12 show that the hydrogen fuel cell power output and battery State of Charge (SOC) are managed within reasonable ranges, demonstrating efficient electrical energy utilization.
  • Improved Adaptability: The DDPG algorithm exhibits high real-time scheduling capability and system adaptability, crucial for handling uncertain flight conditions.

Limitations and Future Work

  • Limitations:
    • The paper acknowledges that while MPC alone cannot fully address system uncertainty, and DDPG alone might not optimize the electrical power side as effectively, their combination achieves optimization. This implies that neither method is perfect in isolation.
  • The study relies on simulated data and scenarios; real-world flight experiments are needed for validation.
  • Future Work:
    • Advanced ML Algorithms: Integrate more sophisticated machine learning techniques like reinforcement learning with attention mechanisms, multi-agent systems, or deep neural networks specifically designed for UAVs.
  • Real-World Flight Experiments: Conduct practical flight tests to validate the framework's effectiveness across various flight missions and environmental conditions (weather, payload, trajectory changes).
  • Industry Collaboration: Partner with industry to integrate the framework into commercial UAV platforms for applications in surveillance, logistics, and environmental monitoring.
  • Broader Applications: Explore applications beyond air-ground-space collaborative power inspection.

Summary updated

BibTeX
@article{li2025energy,
  author = {Haitao Li and Chenyu Wang and Shufu Yuan and Hui Zhu and Bo Li and Yuexin Liu and Li Sun},
  title = {{Energy Scheduling of Hydrogen Hybrid UAV Based on Model Predictive Control and Deep Deterministic Policy Gradient Algorithm}},
  journal = {Algorithms},
  year = {2025},
  doi = {10.3390/a18020080},
}
Figure