Explanation Augmented Feedback in Human-in-the-Loop Reinforcement Learning

Lin Guan, Mudit Verma, Subbarao Kambhampati

Abstract

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.

Introduction

Deep Reinforcement Learning (DRL) agents often require millions of trials to learn even simple tasks, making them notoriously sample-inefficient. To mitigate this, Human-in-the-Loop Reinforcement Learning (HIRL) incorporates human evaluative feedback (e.g., "good" or "bad"). However, this binary signal is "low-bandwidth"—it informs the agent that an action was correct, but fails to explain why. While supervised learning has successfully integrated human "explanations" like feature importance, applying this to RL is challenging due to sparse feedback and dynamic environments.

The EXPAND Framework

To address these limitations, we introduce EXPAND (EXPlanation AugmeNted feeDback). This framework "widens the pipeline" by allowing humans to provide visual explanations—specifically bounding boxes—alongside their evaluative feedback.

Overview of the EXPAND framework, integrating binary evaluative feedback and visual saliency annotations.

EXPAND consists of four primary components:

  1. Advantage-based Feedback Learning: Instead of a separate reward model, EXPAND maps feedback directly to the Advantage function ($A(s, a)$). A "good" label identifies an action as the current best, while a "bad" label penalizes it using a large-margin loss.
  2. Context-Aware Data Augmentation: When a human highlights a salient region (e.g., the ball in Pong), the system generates augmented states by applying Gaussian blur to all non-annotated areas.
  3. Invariance Constraints: An auxiliary loss forces the agent's internal representation to remain invariant to these background perturbations, teaching the model to ignore irrelevant "style" and focus on task-relevant "content."
  4. Semi-Automated Interface: To minimize human effort, the system utilizes object detectors and trackers. A trainer may only need to annotate a single frame, and the system tracks the salient objects throughout the trajectory.

Experimental Evaluation

We evaluated EXPAND across five environments: Pixel-Taxi (a grid-world navigation task) and four Atari games (Pong, Asterix, Montezuma’s Revenge, and Enduro-1000). These tasks offer varying levels of visual complexity and reward sparsity. Testing involved both a synthetic oracle for systematic benchmarking and a user study with real human trainers to account for noisy, real-world feedback.

Key Results and Discussion

EXPAND demonstrated significant improvements in both sample efficiency and final performance across all tested domains.

Performance comparison of EXPAND against baselines across five environments, demonstrating superior sample efficiency and final scores.
  • Sample Efficiency: In Pixel-Taxi and Pong, EXPAND achieved near-optimal policies with 35% fewer environment samples and human signals than standard HIRL baselines.
  • Performance Gains: In the complex Enduro-1000 environment, EXPAND reached a score of 60 within 80k samples, whereas the leading baseline (DQN-TAMER) required 120k samples—a 30% improvement.
  • Robustness: The user study confirmed that EXPAND remains effective even with the sparse and noisy feedback characteristic of human trainers.
  • Ablation Studies: Results showed that the combination of advantage-based loss and invariance constraints is essential; neither technique alone achieves the same level of performance.

Conclusion

By transforming binary feedback into a form of guided feature engineering, EXPAND allows RL agents to learn more effectively from human knowledge. Future work will explore more complex domain-specific perturbations and active learning strategies to further reduce the human workload by querying the trainer only when the agent is most uncertain.

Summary updated

Abstract

Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.
BibTeX
@inproceedings{guan2021explanation,
  author = {Lin Guan and Mudit Verma and Subbarao Kambhampati},
  title = {{Explanation Augmented Feedback in Human-in-the-Loop Reinforcement Learning}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=SEz-FQltAYN},
}
Figure