Providing Uncertainty-Based Advice for Deep Reinforcement Learning Agents (Student Abstract)

Felipe Leno da Silva, Pablo Hernandez-Leal, Bilal Kartal, Matthew E. TaylorAAAI 2020 · OpenReview

Abstract

The sample-complexity of Reinforcement Learning (RL) techniques still represents a challenge for scaling up RL to unsolved domains. One way to alleviate this problem is to leverage samples from the policy of a demonstrator to learn faster. However, advice is normally limited, hence advice should ideally be directed to states where the agent is uncertain on the best action to be applied. In this work, we propose Requesting Confidence-Moderated Policy advice (RCMP), an action-advising framework where the agent asks for advice when its uncertainty is high. We describe a technique to estimate the agent uncertainty with minor modifications in standard value-based RL methods. RCMP is shown to perform better than several baselines in the Atari Pong domain.

Introduction

Reinforcement Learning (RL) is notoriously "sample hungry," often requiring millions of interactions with an environment to learn a competent policy. Action Advice, where a teacher (human or AI) provides the correct action for specific states, is a common technique to mitigate this. However, expert advice is often a scarce resource due to human fatigue or limited communication bandwidth.

Most existing methods provide advice at fixed intervals or based on the teacher's perspective of what is important. This paper introduces RCMP, a framework where agents proactively request advice only when their internal uncertainty is high, ensuring more efficient use of limited expert demonstrations by focusing on the learner's needs.

Methods: Requesting Confidence-Moderated Policy advice (RCMP)

The core of RCMP is a novel way to measure an agent's internal uncertainty within a standard Deep Q-Network (DQN) framework.

Multi-Head Architecture

Inspired by "Bootstrapped DQN," the standard neural network is modified to have multiple "heads" as the final layer. Each head independently estimates the Q-values (expected rewards) for each action. Because each head is initialized differently and experiences slightly different data during training, their predictions vary.

Multi-head neural network architecture for uncertainty estimation.

Uncertainty Metric and Loop

The variance between these heads serves as a proxy for uncertainty. If the heads disagree significantly on the value of a state, the agent is considered "uncertain." The RCMP loop follows these steps:

  1. Calculate Uncertainty: The agent determines its uncertainty $\mu(s)$ for the current state.
  2. Query Advice: If uncertainty is high, it queries a demonstrator and executes the suggested action.
  3. Self-Policy: If uncertainty is low, it follows its own learned policy.
  4. Training: Each head is trained separately using standard RL cost functions, and the final decision-making policy uses the average of all heads.

Results

The researchers evaluated RCMP in the Atari Pong environment using a pre-trained A3C agent as the expert teacher. RCMP was compared against three baselines: No Advice, Random Advice, and Importance Advising.

Training performance comparison in Atari Pong, showing RCMP outperforming other methods.

RCMP achieved higher total rewards than all other methods. While RCMP stops asking for advice once the budget (10,000 instances) is spent, it shows a significant performance jump over other methods starting around 1.5 million learning steps. This demonstrates that estimating agent uncertainty allows for a much more effective use of demonstrated data than providing advice randomly or based solely on teacher heuristics.

Conclusion and Future Work

RCMP successfully repurposes multi-head architectures—originally designed for exploration—to manage a limited advice budget. By focusing advice on states where the learner lacks confidence, it improves sample efficiency. Future work includes exploring alternative uncertainty measures, integrating advice directly into the loss function, and testing scalability in higher-dimensional state spaces.

Summary updated

Abstract

The sample-complexity of Reinforcement Learning (RL) techniques still represents a challenge for scaling up RL to unsolved domains. One way to alleviate this problem is to leverage samples from the policy of a demonstrator to learn faster. However, advice is normally limited, hence advice should ideally be directed to states where the agent is uncertain on the best action to be applied. In this work, we propose Requesting Confidence-Moderated Policy advice (RCMP), an action-advising framework where the agent asks for advice when its uncertainty is high. We describe a technique to estimate the agent uncertainty with minor modifications in standard value-based RL methods. RCMP is shown to perform better than several baselines in the Atari Pong domain.
BibTeX
@inproceedings{silva2020providing,
  author = {Felipe Leno da Silva and Pablo Hernandez-Leal and Bilal Kartal and Matthew E. Taylor},
  title = {{Providing Uncertainty-Based Advice for Deep Reinforcement Learning Agents (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2020},
  url = {https://openreview.net/forum?id=Iw3gZfOfBO},
}
Figure