Abstract
Reinforcement learning (RL) agents often suffer from sample inefficiency, requiring extensive trial-and-error to master complex tasks. While human advice can accelerate learning, it is often constrained by a limited budget. This paper introduces a meta-strategy that leverages human intuition regarding "state criticality"—the importance of an action in a given state—to optimize advice-seeking behavior. By prioritizing advice in critical states, agents can utilize their limited budgets more effectively. Experiments in the Atari Pong environment demonstrate that a multiplicative approach combining agent uncertainty with state criticality significantly outperforms standard advice-seeking methods and baseline RL agents.
Introduction
A primary challenge in Reinforcement Learning (RL) is sample inefficiency. Agents typically learn through millions of interactions, which is often impractical in real-world scenarios. Human-in-the-loop RL attempts to mitigate this by providing expert advice. However, human time is a scarce resource, necessitating an "advice budget."
The fundamental problem lies in advice selection: when should an agent ask for help? Traditional strategies often rely on the agent's internal uncertainty. Unfortunately, early in training, an agent's self-assessment is unreliable; it may waste its budget on trivial states where the choice of action has minimal impact on the outcome. This paper proposes using human-defined "state criticality" as a prior to guide advice-seeking, ensuring the budget is spent where it matters most.
Methodology
The authors propose Criticality-Based Advice, a meta-strategy that can be integrated with existing advice-seeking algorithms. The core concept is State Criticality, a subjective measure provided by a human expert a priori. A state is considered critical if the difference in potential outcomes between the best and worst actions is high.
Two primary methods for integrating criticality were evaluated:
- Logicand Approach (
BDQN-crit1): The agent requests advice only if its internal strategy suggests a need and the state criticality exceeds a certain threshold. - Multiplicative Approach (
BDQN-crit2): The agent's internal uncertainty metric is multiplied by the state's criticality score. If this product exceeds a threshold, advice is requested. This allows high-criticality states to trigger advice even when the agent's uncertainty is moderate.
These strategies were implemented using a Bootstrapped Deep Q-Network (BDQN) as the base learner.
Experimental Evaluation
The proposed methods were tested in the Atari Pong environment with a fixed advice budget of 150,000 instances. The criticality function was derived from prior work, representing human knowledge of which screen regions are most vital for gameplay success.
The performance was compared across four variants:
- BDQN-plain: Standard RL without advice.
- BDQN-adv: Standard uncertainty-based advice seeking.
- BDQN-crit1: Logicand criticality-based advice.
- BDQN-crit2: Multiplicative criticality-based advice.
Results and Discussion
The experimental results highlight the effectiveness of incorporating human intuition into the advice-seeking process. As shown in Figure 1, the multiplicative variant (BDQN-crit2) significantly outperforms all other models, reaching the maximum reward plateau around episode 600.
This suggests that criticality is most effective when used as a weighting factor rather than a simple binary gate. The logicand approach (BDQN-crit1) shows an early advantage over the standard BDQN-adv but is eventually surpassed as the agent's internal uncertainty becomes a more reliable guide in the later stages of training. The results confirm that an agent is often a poor judge of its own needs during early training. By using human-defined criticality as a filter, the agent avoids wasting its budget on low-variance states, preserving resources for high-stakes decisions.
Conclusion and Future Work
This research demonstrates that human-defined state criticality is a powerful tool for improving the efficiency of advice-seeking in RL. The multiplicative integration of criticality and uncertainty provides a robust mechanism for budget management.
Future work should address the manual effort required to design criticality functions, perhaps by exploring methods to learn criticality dynamically. Additionally, testing these strategies across a broader range of environments and underlying advice mechanisms will be crucial for validating the generalizability of the criticality-based meta-strategy.
Summary updated