Representation Learning Enhanced Deep Reinforcement Learning for Optimal Operation of Hydrogen-based Multi-Energy Systems

Zhenyu Pu, Yu Yang, Lun Yang, Qing-Shan Jia, Xiaohong Guan, Costas J. SpanosarXiv 2026 · arXiv

TL;DR

This paper introduces a State-Action Representation Learning Enhanced Deep Reinforcement Learning (SR-DRL) framework to optimize the real-time operation of hydrogen-based multi-energy systems (HMES). By mapping complex, multi-physics system dynamics into a well-structured latent space, the proposed method significantly accelerates learning convergence, slashes operational costs, and prevents critical safety constraint violations.

What Problem Does This Paper Address?

Hydrogen-based multi-energy systems (HMES) are gaining traction as a flexible, low-carbon solution to coordinate electricity, heating, and cooling demands. At the heart of these systems are Hydrogen Energy Storage Systems (HESS)—consisting of electrolyzers, fuel cells, and hydrogen storage tanks.

However, controlling an HMES in real time is incredibly difficult. HESS components exhibit highly nonlinear, multi-physics coupled dynamics (where electrochemical reactions, thermal changes, and pressure variations all interact). Traditional optimization methods like Stochastic Programming or Model Predictive Control (MPC) struggle with the computational burden of solving these multi-stage nonlinear problems under supply and demand uncertainties.

While Deep Reinforcement Learning (DRL) offers a model-free alternative capable of learning from experience, conventional DRL algorithms (like DDPG or TD3) suffer from slow convergence and poor policy quality when applied to complex networked systems. They frequently fail to handle strict operational constraints, leading to safety hazards like hydrogen tank overpressure.

Hydrogen-based multi-energy system for a building community

Methods

The authors tackle these challenges by first building a comprehensive operational model that accurately captures the nonlinear dynamics of HESS, such as power-dependent efficiency curves and thermal balance equations.

To overcome the limitations of standard DRL, they introduce SR-DRL, a framework that embeds representation learning directly into the DRL pipeline. The architecture consists of two primary enhancement modules:

  1. State Embedding ($\Phi$): Maps the high-dimensional, raw system state into a compact latent representation.
  2. State-Action Embedding ($\Psi$): Maps the latent state-action pair to predict the next latent state, effectively learning the underlying transition dynamics within the latent space.
Figure 2: The architecture of representation learning enhanced DRL method

Crucially, the representation learning and DRL policy training are decoupled and alternated using a stop-gradient operation to prevent representation collapse. This framework is versatile and can augment standard actor-critic algorithms; the authors demonstrate this by implementing SR-DDPG and SR-TD3.

Datasets

The framework was evaluated using real-world data from the CityLearn dataset, which provided:

  • Time-of-Use (ToU) electricity market prices.
  • Multi-energy demand profiles (electricity, heating, and cooling) for a building community.
  • Solar irradiance data for photovoltaic and solar thermal generation.

The authors structured the data into three representative cases based on different time periods. For each case, 20 days of data were allocated for training the DRL agents, and 10 days were reserved for testing.

Key Insights and Ideas

  • Reshaping State Space Geometry: Principal Component Analysis (PCA) revealed that conventional DRL operates in a scattered, irregular state space filled with task-irrelevant noise. Representation learning reorganizes this space into a compact, smooth, and cluster-aware geometric structure.
  • Decision-Relevant Clustering: The embedding modules successfully group states into distinct, well-separated clusters that correspond to coherent physical modes of the system, making feature extraction much easier for the actor and critic networks.
  • The Danger of Linear Simplifications: The authors proved that omitting multi-physics constraints (like tank pressure dynamics) yields severe safety violations in practice, underscoring the necessity of their comprehensive nonlinear model.

Key Results

The SR-DRL framework consistently outperformed standard DRL counterparts across all test cases:

  • Higher Returns: SR-TD3 and SR-DDPG improved final episode returns by 11.5% to 17.2% compared to standard TD3 and DDPG.
  • Bridging the Optimality Gap: For Case 1, standard TD3 exhibited a 45.1% performance gap compared to the theoretical optimum (Perfect MPC). SR-TD3 successfully shrank this gap to just 20.9%.
  • Enhanced Safety: Conventional DRL agents frequently violated hydrogen tank pressure limits during testing. In contrast, the SR-DRL methods dramatically reduced constraint violation penalties, maintaining safe boundaries throughout the evaluation period.

Limitations and Future Work

While highly effective, the proposed method depends heavily on extensive historical data or an accurate simulator to properly train the state-action embedding modules and the DRL policy. If the simulator fails to capture real-world anomalies, the learned representations might degrade.

As a next step, the authors suggest extending this representation learning approach to Multi-Agent Deep Reinforcement Learning (MADRL) frameworks to optimize the coordinated operation of multiple interconnected multi-energy microgrids.

Summary updated

Abstract

Hydrogen-based multi-energy systems (HMES) have emerged as a promising low-carbon and energy-efficient solution, as it can enable the coordinated operation of electricity, heating and cooling supply and demand to enhance operational flexibility, improve overall energy efficiency, and increase the share of renewable integration. However, the optimal operation of HMES remains challenging due to the nonlinear and multi-physics coupled dynamics of hydrogen energy storage systems (HESS) (consisting of electrolyters, fuel cells and hydrogen tanks) as well as the presence of multiple uncertainties from supply and demand. To address these challenges, this paper develops a comprehensive operational model for HMES that fully captures the nonlinear dynamics and multi-physics process of HESS. Moreover, we propose an enhanced deep reinforcement learning (DRL) framework by integrating the emerging representation learning techniques, enabling substantially accelerated and improved policy optimization for spatially and temporally coupled complex networked systems, which is not provided by conventional DRL. Experimental studies based on real-world datasets show that the comprehensive model is crucial to ensure the safe and reliable of HESS. In addition, the proposed SR-DRL approaches demonstrate superior convergence rate and performance over conventional DRL counterparts in terms of reducing the operation cost of HMES and handling the system operating constraints. Finally, we provide some insights into the role of representation learning in DRL, speculating that it can reorganize the original state space into a well-structured and cluster-aware geometric representation, thereby smoothing and facilitating the learning process of DRL.
BibTeX
@misc{pu2026representation,
  author = {Zhenyu Pu and Yu Yang and Lun Yang and Qing-Shan Jia and Xiaohong Guan and Costas J. Spanos},
  title = {{Representation Learning Enhanced Deep Reinforcement Learning for Optimal Operation of Hydrogen-based Multi-Energy Systems}},
  year = {2026},
  eprint = {2602.00027},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.00027},
}
Figure