Abstract
This paper introduces a novel framework for enhancing the transparency of AI policies by treating humans as active learners. By applying educational principles such as the Zone of Proximal Development (ZPD) and scaffolding, the proposed closed-loop teaching algorithm selects demonstrations that effectively update a human's understanding of an AI's reward function without exceeding their cognitive capacity.
Introduction
As reinforcement learning (RL) agents become more prevalent in real-world applications, it is essential for humans to understand the underlying reward functions that drive their behavior. While Inverse Reinforcement Learning (IRL) typically models how agents learn from humans, this research explores the inverse: how an AI agent can effectively teach its reward function to a human.
The primary challenge lies in the limited computational capacity of humans. Traditional machine teaching methods often prioritize maximizing mathematical information gain, which can lead to demonstrations that are too complex for humans to reconcile with their existing mental models. If the discrepancy between a human's expectation and the robot's action is too large, the learning process may stall.
Methods: Scaffolding and the Zone of Proximal Development
The proposed framework models the human as an IRL learner and utilizes two key educational concepts:
- Zone of Proximal Development (ZPD): The AI identifies demonstrations that deviate slightly from the human's "counterfactual expectation." By providing information that is "just right"—informative yet manageable—the system ensures the human can successfully update their mental model.
- Scaffolding and Closed-Loop Teaching: Rather than using a static set of examples, the AI employs a particle filter to maintain a probabilistic model of the human's current beliefs. The reward function is decomposed into Knowledge Concepts (KCs), such as the relative cost of different terrains.
The system provides scaffolded demonstrations followed by diagnostic tests. If a human fails a test, the AI provides remedial demonstrations and corrective feedback until the concept is mastered.
Evaluation and Results
The framework was evaluated using grid-world environments, specifically a delivery robot navigating terrains with varying costs (e.g., mud patches). The study compared "open-loop" teaching (pre-selected demonstrations) against the adaptive "closed-loop" algorithm.
Key Insights
- Balancing Informativeness and Difficulty: The most mathematically informative demonstrations are often the most difficult for humans to process. Effective teaching requires balancing these two factors.
- Counterfactual Reasoning: Humans learn most effectively when robot behavior contradicts their current mental model in a clear, manageable way.
- The Testing Effect: Diagnostic tests serve as both a measurement tool and a primary teaching mechanism. Correcting errors on tests is more impactful than passive observation.
Key Results
In user studies, the closed-loop teaching algorithm reduced "regret"—the difference between the human's predicted action and the robot's optimal action—by 41% compared to the baseline. Additionally, users reported significantly higher usability for the adaptive system in complex domains.
Conclusion and Future Work
While successful in grid-world environments, the current approach is limited by the dimensionality of the reward functions, as humans struggle to reason about more than three variables simultaneously. Future research will focus on increasing environment expressivity and integrating these demonstrations with other Explainable AI (XAI) techniques, such as feature highlighting, to further clarify the robot's decision-making process.
Summary updated