Abstract
The sample-complexity of Reinforcement Learning (RL) techniques still represents a challenge for scaling up RL to unsolved domains. One way to alleviate this problem is to leverage samples from the policy of a demonstrator to learn faster. However, advice is normally limited, hence advice should ideally be directed to states where the agent is uncertain on the best action to be applied. In this work, we propose Requesting Confidence-Moderated Policy advice (RCMP), an action-advising framework where the agent asks for advice when its uncertainty is high. We describe a technique to estimate the agent uncertainty with minor modifications in standard value-based RL methods. RCMP is shown to perform better than several baselines in the Atari Pong domain.
Introduction
Reinforcement Learning (RL) is notoriously "sample hungry," often requiring millions of interactions with an environment to learn a competent policy. Action Advice, where a teacher (human or AI) provides the correct action for specific states, is a common technique to mitigate this. However, expert advice is often a scarce resource due to human fatigue or limited communication bandwidth.
Most existing methods provide advice at fixed intervals or based on the teacher's perspective of what is important. This paper introduces RCMP, a framework where agents proactively request advice only when their internal uncertainty is high, ensuring more efficient use of limited expert demonstrations by focusing on the learner's needs.
Methods: Requesting Confidence-Moderated Policy advice (RCMP)
The core of RCMP is a novel way to measure an agent's internal uncertainty within a standard Deep Q-Network (DQN) framework.
Multi-Head Architecture
Inspired by "Bootstrapped DQN," the standard neural network is modified to have multiple "heads" as the final layer. Each head independently estimates the Q-values (expected rewards) for each action. Because each head is initialized differently and experiences slightly different data during training, their predictions vary.
Uncertainty Metric and Loop
The variance between these heads serves as a proxy for uncertainty. If the heads disagree significantly on the value of a state, the agent is considered "uncertain." The RCMP loop follows these steps:
- Calculate Uncertainty: The agent determines its uncertainty $\mu(s)$ for the current state.
- Query Advice: If uncertainty is high, it queries a demonstrator and executes the suggested action.
- Self-Policy: If uncertainty is low, it follows its own learned policy.
- Training: Each head is trained separately using standard RL cost functions, and the final decision-making policy uses the average of all heads.
Results
The researchers evaluated RCMP in the Atari Pong environment using a pre-trained A3C agent as the expert teacher. RCMP was compared against three baselines: No Advice, Random Advice, and Importance Advising.
RCMP achieved higher total rewards than all other methods. While RCMP stops asking for advice once the budget (10,000 instances) is spent, it shows a significant performance jump over other methods starting around 1.5 million learning steps. This demonstrates that estimating agent uncertainty allows for a much more effective use of demonstrated data than providing advice randomly or based solely on teacher heuristics.
Conclusion and Future Work
RCMP successfully repurposes multi-head architectures—originally designed for exploration—to manage a limited advice budget. By focusing advice on states where the learner lacks confidence, it improves sample efficiency. Future work includes exploring alternative uncertainty measures, integrating advice directly into the loss function, and testing scalability in higher-dimensional state spaces.
Summary updated