Delay-Aware Reinforcement Learning: Insights From Delay Distributional Perspective
Lists
Only you see this
Abstract
Although deep reinforcement learning (DRL) has achieved great success across various domains, the presence of random delays in real-world scenarios (e.g., remote control) poses a significant challenge to its practicality. Existing delay-aware DRLs mainly focus on state augmentation with historical memory, ensuring that the actions taken are aligned with the true state. However, these approaches still rely on the conventional expected $Q$ value. In contrast, to model delay uncertainty, we aim to go beyond the expected value and propose a distributional DRL to represent the distribution of this $Q$ value. Based on the delay distribution, we further propose a correction mechanism for the distributional $Q$ value, enabling the agent to learn accurate returns in delayed environments. Finally, we apply these techniques to design the delay-aware distributional actor-critic (DADAC) DRL framework, in which the critic is the corrected distributional value function. Experimental results demonstrate that compared to the state-of-the-art delay-aware DRL methods, the proposed DADAC exhibits substantial performance advantages in handling random delays in the MuJoCo continuous control tasks. The corresponding source code is available at https://anonymous.4open.science/r/DADAC.
BibTeX
@misc{yu2025delay,
author = {Zhuoru Yu and Chenchen Fu and Hengkai Zhong and Wanyuan Wang and Weiwei Wu and Chun Jason Xue},
title = {{Delay-Aware Reinforcement Learning: Insights From Delay Distributional Perspective}},
year = {2025},
url = {https://openreview.net/forum?id=Y9cVrdYn10},
howpublished = {OpenReview},
note = {Submitted to ICLR 2025},
}Loading related content...