Offline Reward Inference on Graph: A New Thinking
Lists
Only you see this
Abstract
In offline reinforcement learning, reward inference is the key to learning effective policies in practical scenarios. Due to the expensive or unethical nature of environmental interactions in domains such as healthcare and robotics, reward functions are rarely accessible, and the task of inferring rewards becomes challenging. To address this issue, our research focuses on developing a reward inference method that capitalizes on a constrained number of human reward annotations to infer rewards for unlabelled data. Initially, we leverage both the available data and limited reward annotations to construct a reward propagation graph, wherein the edge weights incorporate various influential factors pertaining to the rewards. Subsequently, we employ the constructed graph for transductive reward inference, thereby estimating rewards for unlabelled data. Furthermore, we establish the existence of a fixed point during several iterations of the transductive inference process and demonstrate its at least convergence to a local optimum. Empirical evaluations on locomotion and robotic manipulation tasks substantiate the efficacy of our approach, wherein the utilization of our inferred rewards yields substantial performance enhancements within the offline reinforcement learning framework, particularly when confronted with limited reward annotations.
BibTeX
@misc{qu2024offline,
author = {Bohao Qu and Xiaofeng Cao and Yonggang Zhang and Hechang Chen and Qing Guo and Yi Chang},
title = {{Offline Reward Inference on Graph: A New Thinking}},
year = {2024},
url = {https://openreview.net/forum?id=Cqrv7Sve7g},
howpublished = {OpenReview},
note = {Submitted to ICLR 2024},
}Loading related content...