Better Apprenticeship Learning with LLM Explanations

R. Grover, Aryan Vats, Nina Moorman, Anupam Agrawal, Matthew GombolayAAAI 2024 · DOI

TL;DR

This paper introduces a novel framework that uses Large Language Models (LLMs) to explain the decision-making process of a robot learning from human demonstrations. By translating complex "Shapley values" into natural language, the system helps non-expert users understand how to provide better corrective demonstrations, ultimately improving the robot's learned behavior.

What Problem Does This Paper Address?

As robots become more integrated into our lives, particularly in caregiving roles, they need to adapt to the evolving needs and preferences of their users. This adaptability is challenging to pre-program. A promising approach is for robots to learn directly from human demonstrations. However, humans are imperfect demonstrators, and they often struggle to understand why a robot isn't performing as expected or how to correct its behavior. This is especially true for non-expert users, like older adults receiving care. Existing methods for explaining robot behavior, like feature importance maps, can be too technical. This paper addresses the gap in providing understandable, actionable feedback to non-expert demonstrators, enabling them to effectively guide robot learning.

Methods

The core of the proposed framework lies in bridging the gap between a robot's learned policy and a human demonstrator's understanding. Here's how it works:

  1. Inverse Reinforcement Learning (IRL): The robot first learns a reward function from initial human demonstrations using a technique called Maximum Entropy Deep Inverse Reinforcement Learning (Deep MaxEnt IRL). This allows the robot to infer the underlying reward structure that explains the observed behavior.
  2. Shapley Additive Explanations (SHAP): To understand why the robot makes certain decisions, the framework uses SHAP. SHAP is a model-agnostic technique that calculates the contribution of each feature to a specific prediction, essentially assigning an "importance score" (Shapley value) to each factor influencing the robot's action.
  3. Large Language Model (LLM) for Explanation: This is the novel contribution. The calculated Shapley values, along with contextual information about the environment and task, are fed into an LLM (specifically, Google DeepMind's Gemini 1.5 Flash). The LLM then generates a human-interpretable explanation of these Shapley values. This explanation not only clarifies what each feature means but also suggests how the demonstrator might need to adjust their future demonstrations to improve the robot's policy.
  4. Corrective Demonstrations: Based on the LLM's explanation, the demonstrator can provide new, corrective demonstrations. The system then retrains the robot's policy with these improved demonstrations.

Datasets

The paper uses the Wumpus World environment for evaluation. This environment is presented in both deterministic and stochastic versions. While not a real-world dataset, it serves as a controlled benchmark to test the framework's ability to handle varying levels of complexity and uncertainty. The "dataset" for training consists of human trajectories (demonstrations) provided by participants.

Key Insights and Ideas

  • LLMs as Interpreters for AI: The paper highlights the potential of LLMs not just for generating text, but for acting as sophisticated interpreters of complex AI model outputs, making them accessible to non-experts.
  • Bridging the "Explainability Gap": Simply knowing that a feature is important (via Shapley values) isn't enough. The LLM's ability to translate why it's important and how to act on that information is crucial for effective human-robot collaboration.
  • Empowering Non-Expert Demonstrators: The framework aims to democratize robot learning by enabling individuals without deep technical expertise to actively participate in shaping robot behavior, which is vital for personalized assistive technologies.
  • Iterative Improvement: The process is designed to be iterative, allowing for continuous refinement of the robot's policy through repeated cycles of explanation and corrective demonstration.

Key Results

The paper reports on a proof-of-concept pilot study where the authors themselves acted as demonstrators.

  • Performance Improvement: Across both deterministic and stochastic Wumpus World environments, the framework demonstrated significant performance improvements after one or two rounds of corrective feedback.
  • Win Rate: In the deterministic environment, a 100% win rate was achieved after just one round of feedback. In more challenging stochastic environments, win rates improved from around 58% to 83.5% after one round, and further to 66.56% after two rounds (though the stochastic results show some variability).
  • Feature Convergence: Violin plots showed that the learned feature weights (represented by SHAP values) converged across deterministic and stochastic environments after the second iteration, indicating the pipeline effectively guides the agent towards learning similarly weighted features. For example, the "distance to goal" feature became more negative (rewarding closeness), and "distance to monster" became more positive (rewarding avoidance).

Limitations and Future Work

  • Pilot Study Participants: The initial study involved the authors as participants, not representative non-expert users. Future work will involve user studies with actual care providers and receivers.
  • Environment Specificity: While Wumpus World is a useful testbed, its direct applicability to caregiving tasks is limited. Future work will evaluate the framework in more complex, real-world environments like RCareWorld.
  • Prompt Engineering: The authors acknowledge that further prompt engineering for the LLM could improve the interaction and explanations.
  • Sampling Strategy: The method for sampling demonstrations for retraining could be refined, potentially by weighting more recent demonstrations higher.
  • Baseline Comparisons: Future work will include comparisons against baselines that use only Shapley values or no explanations at all to better quantify the LLM's contribution.
  • Generalizability: Evaluating the framework in partially observable and continuous environments is also a future direction.

Summary updated

Abstract

As the population ages, care robots will play an increasing role in assisting caregiving by taking on repetitive or physically cumbersome activities. To effectively provide care, robotic agents must be able to meet the needs and preferences of care receivers. However, these needs and preferences may change over time, making it intractable to pre-define the way the care robot should act before deployment. Instead, the care robot should be able to learn directly from non-expert end-user demonstrations. However, prior work investigating the feasibility of learning a policy from older adult demonstrations finds that older adult demonstrators desire a better understanding of what the robot needs them to do, and how. To help demonstrators understand how to improve on suboptimal or heterogeneous demonstrations, we propose to utilize a Large Language Model to provide human-interpretable explanations of Shapley values of a policy. These explanations enable the demonstrator to understand how the policy is performing, and what changes are needed, informing their corrective demonstrations. We showcase our framework's performance in deterministic and stochastic versions of Wumpus World.
BibTeX
@inproceedings{grover2024better,
  author = {R. Grover and Aryan Vats and Nina Moorman and Anupam Agrawal and Matthew Gombolay},
  title = {{Better Apprenticeship Learning with LLM Explanations}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2024},
  doi = {10.1609/aaaiss.v4i1.31783},
}
Figure