A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning
Lists
Only you see this
Abstract
Distributional Reinforcement Learning (RL) estimates return distribution mainly by learning quantile values via minimizing the quantile Huber loss function, entailing a threshold parameter often selected heuristically or via hyperparameter search, which may not generalize well and can be suboptimal. This paper introduces a generalized quantile Huber loss function derived from Wasserstein distance (WD) calculation between Gaussian distributions, capturing noise in predicted (current) and target (Bellman-updated) quantile values. Compared to the classical quantile Huber loss, this innovative loss function enhances robustness against outliers. Notably, the classical Huber loss function can be seen as an approximation of our proposed loss, enabling parameter adjustment by approximating the amount of noise in the data during the learning process. Empirical tests on Atari games, a common application in distributional RL, and a recent hedging strategy using distributional RL, validate the effectiveness of our proposed loss function and its potential for parameter adjustments in distributional RL. The implementation of the proposed loss function is available here.
BibTeX
@inproceedings{malekzadeh2024robust,
author = {Parvin Malekzadeh and Konstantinos N. Plataniotis and Zissis Poulos and Zeyu Wang},
title = {{A Robust Quantile Huber Loss With Interpretable Parameter Adjustment In Distributional Reinforcement Learning}},
booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
year = {2024},
url = {https://openreview.net/forum?id=oMJhePnawO},
}Loading related content...