Reinforcement Learning for Charging Optimization of Inhomogeneous Dicke Quantum Batteries

Xiaobin Song, Siyuan Bai, Da-Wei Wang, Hanxiao Tao, Xizhe Wang, Rebing Wu, Benben JiangarXiv 2025 · DOI · arXiv

Abstract

Quantum batteries (QBs) offer the potential for "supercharging" through collective quantum effects, but practical implementation is hindered by system inhomogeneity and partial observability. This paper investigates the use of the Soft Actor-Critic (SAC) reinforcement learning algorithm to optimize charging protocols for an inhomogeneous Dicke quantum battery. We demonstrate that by incorporating second-order quantum correlations into the agent's observations, RL can recover near-optimal performance (94%–98% of the full-state baseline) even without access to the complete quantum state. Our findings highlight the importance of multi-body correlations in overcoming parity symmetry constraints and show that RL agents can learn sophisticated, non-myopic strategies to maximize extractable work.

Introduction

Quantum batteries promise significantly faster charging rates than classical systems by leveraging entanglement and collective dynamics. However, real-world physical realizations, such as superconducting qubits, are rarely identical. This "inhomogeneity" or static disorder breaks the symmetries relied upon by many theoretical models. Furthermore, experimentalists typically lack access to the full quantum state ($|\psi\rangle$), operating instead in a Partially Observable Markov Decision Process (POMDP) where only specific observables are measurable. This study explores whether model-free Reinforcement Learning (RL) can navigate these complexities to find efficient charging protocols.

Methodology

We model an inhomogeneous Dicke quantum battery consisting of $N=4$ two-level systems (TLSs) coupled to a common cavity charger. The optimization task is framed as an RL problem using the Soft Actor-Critic (SAC) algorithm, which promotes exploration by maximizing both reward and policy entropy.

Control and Reward Shaping

The agent controls the piecewise-constant coupling strength ($\lambda_c$) between the battery and the charger. To address the challenge of sparse rewards, we employ a scheduled reward function: early training rewards simple energy accumulation, while later stages prioritize ergotropy—the maximum work extractable from the battery via unitary operations.

Information Hierarchy

To evaluate the impact of partial observability, we tested four levels of information availability:

  1. Full quantum state: The theoretical upper bound.
  2. Individual TLS energies: Basic local observables.
  3. Energies + first-order collective averages: Local means.
  4. Energies + first-order averages + second-order correlations: Including two-body relationships.

Results and Discussion

The Role of Quantum Correlations

A critical insight from our experiments is that first-order averages provide negligible benefit over simple energy measurements. This is due to the system's discrete parity symmetry; because the initial state is a parity eigenstate, the expectation values of odd operators (like $\sigma_x$) remain zero, offering no guidance to the agent. However, second-order correlations (even operators) carry essential information about multi-body quantum states, allowing the agent to bridge the performance gap to the full-state baseline.

Mean ergotropy and confidence intervals for different environments under varying levels of available information.

Non-Myopic Charging Strategies

The RL agent demonstrates sophisticated behavior by not simply chasing the highest instantaneous energy. Instead, it learns to navigate plateaus or temporary energy declines to reach a state that allows for higher terminal ergotropy. This suggests the agent is "waiting" for specific quantum interference conditions that maximize extractable work.

Learned coupling schedules (λ_c) and the resulting ergotropy over time for different information levels.

Key Findings

  • Performance Recovery: Agents utilizing second-order correlations achieved between 94% and 98% of the performance of the full-state baseline across three different disordered environments (Env1, Env2, Env3).
  • Robustness: While a small gap remains compared to full-state access, the hierarchy of performance remains consistent across different realizations of static disorder, proving the robustness of the correlation-augmented approach.
  • Efficiency: Full observability leads to the fastest convergence, but partial observability with second-order correlations is a viable and experimentally grounded alternative.

Conclusion and Future Work

This work demonstrates that RL is a powerful tool for optimizing "messy" quantum systems. While the current study focused on $N=4$ qubits, future research will address scalability challenges, as the number of second-order correlations grows quadratically with system size. Potential solutions include the use of Recurrent Neural Networks (RNNs) to build internal state representations or compressed sensing techniques to manage high-dimensional observation spaces. Additionally, exploring RL's performance under dynamic noise and decoherence remains a high-priority area for practical QB development.

Summary updated

BibTeX
@misc{song2025reinforcement,
  author = {Xiaobin Song and Siyuan Bai and Da-Wei Wang and Hanxiao Tao and Xizhe Wang and Rebing Wu and Benben Jiang},
  title = {{Reinforcement Learning for Charging Optimization of Inhomogeneous Dicke Quantum Batteries}},
  year = {2025},
  eprint = {2511.12176},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2511.12176},
}
Figure