Predictive Estimation for Reinforcement Learning with Time-Varying Reward Functions

Abolfazl Hashemi, Antesh Upadhyay

Abstract

The Adversarial Markov Decision Process (MDP) serves as a framework for exploring unknown and time-varying tasks within the framework of Reinforcement Learning (RL). However, it tends to yield pessimistic results when characterized by the concept of regret. This is primarily because the adversarial MDP framework assumes that the loss function can change arbitrarily from one episode to another. In contrast, many real-world scenarios exhibit a more predictable evolution of loss functions rather than an adversarial one. To tackle this concern, we introduce a novel model. In this new model, the primary objective is to develop a policy that minimizes its regret (or equivalently, maximizes its reward) by utilizing a sequence of predictive estimators for the loss functions. Additionally, we put forth a policy optimization approach based on the optimistic mirror descent algorithm. This method yields an optimistic expected regret, which gracefully decreases in magnitude as the predictive estimators' accuracy and estimation power improve.
BibTeX
@inproceedings{hashemi2023predictive,
  author = {Abolfazl Hashemi and Antesh Upadhyay},
  title = {{Predictive Estimation for Reinforcement Learning with Time-Varying Reward Functions}},
  booktitle = {ACSSC},
  year = {2023},
  doi = {10.1109/ieeeconf59524.2023.10477059},
}

Loading related content...

Figure