The Adversarial Regulation of the Temporal Difference Loss Costs More Than Expected

Ezgi KorkmazICLR 2023 · OpenReview

Abstract

Deep reinforcement learning research has enabled reaching significant performance levels for sequential decision making in MDPs with highly complex observations and state dynamics with the aid of deep neural networks. However, this aid came with a cost that is inherent to deep neural networks which have increased sensitivities towards indistinguishable peculiarly crafted non-robust directions. To alleviate these sensitivities several studies suggested techniques to cope with this problem via explicitly regulating the temporal difference loss for the worst-case sensitivity. In our study, we show that these worst-case regularization techniques come with a cost that intriguingly causes inconsistencies and overestimations in the state-action value functions. Furthermore, our results essentially demonstrate that vanilla trained deep reinforcement learning policies have more accurate and consistent estimates for the state-action values. We believe our results reveal foundational intrinsic properties of the adversarial training techniques and demonstrate the need to rethink the approach to robustness in deep reinforcement learning.
BibTeX
@inproceedings{korkmaz2023adversarial,
  author = {Ezgi Korkmaz},
  title = {{The Adversarial Regulation of the Temporal Difference Loss Costs More Than Expected}},
  booktitle = {International Conference on Learning Representations},
  year = {2023},
  url = {https://openreview.net/forum?id=vDybC2brXKh},
}

Loading related content...

Figure