Abstract
Human explanation (e.g., in terms of feature importance) has been recently used to extend the communication channel between human and agent in interactive machine learning. Under this setting, human trainers provide not only the ground truth but also some form of explanation. However, this kind of human guidance was only investigated in supervised learning tasks, and it remains unclear how to best incorporate this type of human knowledge into deep reinforcement learning. In this paper, we present the first study of using human visual explanations in human-in-the-loop reinforcement learning (HRL). We focus on the task of learning from feedback, in which the human trainer not only gives binary evaluative "good" or "bad" feedback for queried state-action pairs, but also provides a visual explanation by annotating relevant features in images. We propose EXPAND (EXPlanation AugmeNted feeDback) to encourage the model to encode task-relevant features through a context-aware data augmentation that only perturbs irrelevant features in human salient information. We choose five tasks, namely Pixel-Taxi and four Atari games, to evaluate the performance and sample efficiency of this approach. We show that our method significantly outperforms methods leveraging human explanation that are adapted from supervised learning, and Human-in-the-loop RL baselines that only utilize evaluative feedback.
Introduction
Deep Reinforcement Learning (DRL) has achieved remarkable success in complex domains, yet it remains notoriously sample inefficient, often requiring millions of environment interactions. Human-in-the-Loop Reinforcement Learning (HIRL) aims to mitigate this by incorporating human knowledge, typically through binary evaluative feedback (e.g., "good" or "bad"). However, binary feedback represents a "thin" communication channel; it informs the agent that an action was correct or incorrect, but fails to explain why. This lack of context can lead the agent to learn spurious correlations, such as associating a successful maneuver with irrelevant background features.
To address this, we introduce EXPAND (EXPlanation AugmeNted feeDback), a framework that "widens the pipeline" by allowing human trainers to provide visual explanations alongside evaluative feedback. By highlighting salient objects, humans can guide the agent's attention toward task-critical features, significantly accelerating the learning process.
The EXPAND Framework
The core of EXPAND lies in its ability to integrate high-level human visual saliency into the low-level pixel-based learning process of a DRL agent.
The framework utilizes two primary mechanisms:
- Advantage-Based Feedback: Human binary feedback is mapped to an advantage function. Positive feedback encourages the model to treat the selected action as optimal, while negative feedback penalizes the action relative to alternatives.
- Context-Aware Data Augmentation: When a trainer provides a visual explanation—for instance, by drawing bounding boxes around a passenger in Pixel-Taxi or the ball in Pong—the system generates augmented states. These states apply Gaussian blurring to all regions outside the human-annotated boxes.
- Invariance Constraints: An auxiliary loss function forces the agent's internal representations to be invariant to these perturbations. This explicitly instructs the model to ignore "noisy" background information and focus on the clear, human-identified salient features.
To minimize human workload, the system incorporates off-the-shelf object trackers. A trainer only needs to annotate a few keyframes, and the system propagates these explanations across the trajectory.
Experimental Evaluation
We evaluated EXPAND across five diverse environments: Pixel-Taxi, a discrete grid-world task, and four Atari games: Pong, Asterix, Enduro-1000, and Montezuma’s Revenge (Level 1). Performance was measured against several baselines, including standard DQN, DQN-TAMER (which uses human rewards), and DQN-Feedback (which uses binary evaluative feedback without explanations).
The study utilized both a synthetic oracle for systematic benchmarking and a user study involving three graduate students to assess real-world human-agent interaction.
Key Results
Our experimental results demonstrate that EXPAND significantly enhances both sample efficiency and final performance across all tested domains.
Key findings include:
- Efficiency Gains: In tasks like Pixel-Taxi and Pong, EXPAND reached near-optimal policies using approximately 35% fewer environment samples and human signals than binary-only feedback methods.
- Robustness to Noise: By using explanations to drive data augmentation rather than directly forcing attention maps (a common but unstable approach in RL), EXPAND provides a more stable training signal.
- Human-Centric Success: In the Enduro-1000 user study, EXPAND achieved higher scores than binary-only feedback, even though humans provided roughly 15% fewer total feedback signals due to the time required for annotation. This indicates that the high information density of explanations more than compensates for the increased effort per signal.
Conclusion and Future Work
EXPAND demonstrates that "widening the pipeline" with visual explanations is a powerful strategy for making HIRL more practical and efficient. By bridging the gap between human symbolic reasoning and neural network representations through context-aware augmentation, we enable agents to learn the "right" features faster.
Future research will explore more sophisticated perturbation types beyond Gaussian blurring and the integration of active learning to allow the agent to query the human trainer more strategically when uncertainty is high. Scaling these techniques to high-fidelity 3D environments and real-world robotics remains a key objective.
Summary updated