Reducing Reward Dependence in RL Through Adaptive Confidence Discounting

Muhammed Yusuf Satici, David L. RobertsarXiv 2025 · DOI · arXiv

Abstract

Reinforcement Learning (RL) typically relies on frequent external reward signals, which can be prohibitively expensive in real-world applications. This paper introduces a framework that utilizes an internal reward model and an Adaptive Confidence Discounting mechanism to minimize environment feedback. By requesting rewards only when the agent's confidence is low, the proposed method achieves baseline-level performance with as little as 20% of the total available rewards across various tasks, including robotics and grid-world environments.

Introduction

In complex RL scenarios, such as human-in-the-loop training or high-fidelity scientific simulations, the cost of evaluating an action (the reward signal) often outweighs the cost of the interaction itself. While traditional "sparse reward" research focuses on exploration, this work addresses feedback efficiency. The goal is to enable agents to "guess" rewards using a learned internal model and selectively query the environment for ground truth only when necessary.

Methodology: Adaptive Confidence Discounting

The proposed architecture centers on a reward function model—a secondary neural network trained to predict rewards from state-action pairs. The decision to request external feedback is governed by three main components:

  1. Confidence Estimation: The agent calculates confidence using the harmonic mean of Action Entropy (AE) (policy uncertainty) and Reward Entropy (RE) (model uncertainty). This dual-entropy approach ensures the agent remains "risk-averse" regarding its internal knowledge.
  2. Regularization: To prevent overconfidence in a potentially biased internal model, the authors implement Exponential and Hyperbolic regularization. These terms artificially decay confidence over time since the last external reward, forcing periodic "check-ins" with the environment.
  3. Dual-Buffer Training: Standard RL training uses a primary Replay Buffer, while the reward model is trained exclusively on ground-truth data from a dedicated Feedback Buffer.

Key Insights

Model Size and Overfitting

A counter-intuitive finding was that smaller reward models often outperformed larger ones. Large models tended to overfit to the initial sparse rewards, leading to premature overconfidence and a cessation of environment queries. This suggests that for reward modeling, simplicity can be a safeguard against misplaced certainty.

Natural Curriculum Learning

The system inherently creates a "soft curriculum." Early in training, high uncertainty leads to frequent reward requests. As the agent masters the state space and its internal model improves, it naturally "weans" itself off external feedback, focusing its queries on novel or complex states.

Experimental Results

The framework was evaluated on three benchmarks: Key-Lock (discrete), Fetch-Push (continuous robotics), and Parking (dense reward).

Reward Efficiency in Robotics

In the Fetch-Push task, the proposed algorithm matched the performance of Hindsight Experience Replay (HER) while reducing the number of external rewards by over 75%. As shown in the figure below, the AE+RE variants achieve comparable scores with significantly less environment feedback.

Comparison of reward efficiency in the Fetch-Push environment. The AE+RE variants achieve scores comparable to HER with a fraction of the feedback.

Asymptotic Performance and Scalability

Across all environments, the "AE + RE" method demonstrated superior feedback efficiency compared to standard baselines like DQN and A2C. In the Parking environment, feedback dependency was reduced by 95% (from 431k to 24k rewards) without sacrificing final performance. The following figure summarizes the performance gains and the impact of model architecture.

Comprehensive performance metrics: (a) Asymptotic performance, (b) Reward requirements across baselines, and (c) Impact of model size on learning.

Conclusion and Future Work

Adaptive Confidence Discounting provides a robust path toward feedback-efficient RL. While the method shows significant promise, future work is needed to address hyperparameter sensitivity and potential local optima where an agent might become stuck in a confident but incorrect internal model. Investigating the spatial distribution of reward requests could further illuminate which environmental features are most challenging to model.

Summary updated

BibTeX
@misc{satici2025reducing,
  author = {Muhammed Yusuf Satici and David L. Roberts},
  title = {{Reducing Reward Dependence in RL Through Adaptive Confidence Discounting}},
  year = {2025},
  eprint = {2502.21181},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2502.21181},
}
Figure