Criticality-Based Advice in Reinforcement Learning (Student Abstract)

Yitzhak Spielberg, Amos Azaria

Abstract

Reinforcement learning (RL) agents often suffer from sample inefficiency, requiring extensive trial-and-error to master complex tasks. While human advice can accelerate learning, it is often constrained by a limited budget. This paper introduces a meta-strategy that leverages human intuition regarding "state criticality"—the importance of an action in a given state—to optimize advice-seeking behavior. By prioritizing advice in critical states, agents can utilize their limited budgets more effectively. Experiments in the Atari Pong environment demonstrate that a multiplicative approach combining agent uncertainty with state criticality significantly outperforms standard advice-seeking methods and baseline RL agents.

Introduction

A primary challenge in Reinforcement Learning (RL) is sample inefficiency. Agents typically learn through millions of interactions, which is often impractical in real-world scenarios. Human-in-the-loop RL attempts to mitigate this by providing expert advice. However, human time is a scarce resource, necessitating an "advice budget."

The fundamental problem lies in advice selection: when should an agent ask for help? Traditional strategies often rely on the agent's internal uncertainty. Unfortunately, early in training, an agent's self-assessment is unreliable; it may waste its budget on trivial states where the choice of action has minimal impact on the outcome. This paper proposes using human-defined "state criticality" as a prior to guide advice-seeking, ensuring the budget is spent where it matters most.

Methodology

The authors propose Criticality-Based Advice, a meta-strategy that can be integrated with existing advice-seeking algorithms. The core concept is State Criticality, a subjective measure provided by a human expert a priori. A state is considered critical if the difference in potential outcomes between the best and worst actions is high.

Two primary methods for integrating criticality were evaluated:

  1. Logicand Approach (BDQN-crit1): The agent requests advice only if its internal strategy suggests a need and the state criticality exceeds a certain threshold.
  2. Multiplicative Approach (BDQN-crit2): The agent's internal uncertainty metric is multiplied by the state's criticality score. If this product exceeds a threshold, advice is requested. This allows high-criticality states to trigger advice even when the agent's uncertainty is moderate.

These strategies were implemented using a Bootstrapped Deep Q-Network (BDQN) as the base learner.

Experimental Evaluation

The proposed methods were tested in the Atari Pong environment with a fixed advice budget of 150,000 instances. The criticality function was derived from prior work, representing human knowledge of which screen regions are most vital for gameplay success.

The performance was compared across four variants:

  • BDQN-plain: Standard RL without advice.
  • BDQN-adv: Standard uncertainty-based advice seeking.
  • BDQN-crit1: Logicand criticality-based advice.
  • BDQN-crit2: Multiplicative criticality-based advice.

Results and Discussion

The experimental results highlight the effectiveness of incorporating human intuition into the advice-seeking process. As shown in Figure 1, the multiplicative variant (BDQN-crit2) significantly outperforms all other models, reaching the maximum reward plateau around episode 600.

Learning curves for different advice-seeking strategies in Atari Pong. The multiplicative approach (BDQN-crit2) shows the fastest convergence and highest reward stability.

This suggests that criticality is most effective when used as a weighting factor rather than a simple binary gate. The logicand approach (BDQN-crit1) shows an early advantage over the standard BDQN-adv but is eventually surpassed as the agent's internal uncertainty becomes a more reliable guide in the later stages of training. The results confirm that an agent is often a poor judge of its own needs during early training. By using human-defined criticality as a filter, the agent avoids wasting its budget on low-variance states, preserving resources for high-stakes decisions.

Conclusion and Future Work

This research demonstrates that human-defined state criticality is a powerful tool for improving the efficiency of advice-seeking in RL. The multiplicative integration of criticality and uncertainty provides a robust mechanism for budget management.

Future work should address the manual effort required to design criticality functions, perhaps by exploring methods to learn criticality dynamically. Additionally, testing these strategies across a broader range of environments and underlying advice mechanisms will be crucial for validating the generalizability of the criticality-based meta-strategy.

Summary updated

Abstract

One of the ways to make reinforcement learning (RL) more efficient is by utilizing human advice. Because human advice is expensive, the central question in advice-based reinforcement learning is, how to decide in which states the agent should ask for advice. To approach this challenge, various advice strategies have been proposed. Although all of these strategies distribute advice more efficiently than naive strategies, they rely solely on the agent's estimate of the action-value function, and therefore, are rather inefficient when this estimate is not accurate, in particular, in the early stages of the learning process. To address this weakness, we present an approach to advice-based RL, in which the human’s role is not limited to giving advice in chosen states, but also includes hinting a-priori, before the learning procedure, in which sub-domains of the state space the agent might require more advice. For this purpose we use the concept of critical: states in which choosing the proper action is more important than in other states.
BibTeX
@inproceedings{spielberg2022criticality,
  author = {Yitzhak Spielberg and Amos Azaria},
  title = {{Criticality-Based Advice in Reinforcement Learning (Student Abstract)}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21665},
}
Figure