Inducing Structure in Reward Learning by Learning Features

Andreea Bobu, Marius Wiggert, Claire J. Tomlin, Anca D. DraganInt. J. Robotics Res 2022 · DOI · OpenReview

TL;DR

This paper introduces a "divide and conquer" approach to reward learning for robots. Instead of learning features and reward functions simultaneously from scratch, they propose learning features first using a novel "feature trace" input, and then learning the reward. This method significantly improves sample efficiency and generalization, especially in complex scenarios.

What Problem Does This Paper Address?

Robots learning from human input (reward learning) is crucial for creating adaptable and intuitive systems. Traditional methods often rely on hand-crafted features, which are difficult to define comprehensively for real-world tasks. Modern deep Inverse Reinforcement Learning (IRL) methods learn rewards directly from raw sensor data, but this requires vast amounts of data to implicitly learn relevant features and their combinations. This paper addresses the challenge of learning complex reward functions efficiently and effectively, particularly when the relevant features are not known a priori. The core problem is the high sample complexity and poor generalization of existing methods when faced with learning both features and rewards from scratch.

Methods

The paper proposes a Feature Expansive Reward Learning (FERL) framework. The key innovation is a divide and conquer strategy:

  1. Feature Learning:
    • Novel Human Input: Feature Traces: This is a sequence of states where a specific feature's value monotonically decreases. Humans guide the robot through states from high to low feature expression. This input is designed to be intuitive and less burdensome than providing explicit feature values.
  2. Feature Learning Algorithm: A neural network is trained to predict the monotonic decrease along feature traces and to ensure that traces start and end at consistent high and low feature values, respectively. This algorithm leverages the structure of feature traces to learn complex, non-linear features from high-dimensional state spaces.
  3. Reward Learning: Once features are learned, they are integrated into existing reward learning frameworks. The paper demonstrates this in two settings:
  4. Offline FERL: Features are learned first, then demonstrations are used to learn the reward function that combines these features.
  5. Online FERL: The robot has an initial feature set. If it detects that human input cannot be explained by its current features, it prompts the human to teach a new feature using feature traces, updates its feature set, and then refines the reward.

Datasets

  • Experimental Setup: The experiments were conducted on a physical 7-DOF JACO robotic arm and in a simulated 7-DOF JACO robotic arm environment.
  • Data Collection:
    • Expert Traces: Collected by an expert familiar with the algorithm.
  • User Traces: Collected from non-expert users in a simulated environment.
  • Demonstrations: Collected for both offline and online reward learning.
  • Benchmarks: The paper compares its method against a deep Maximum Entropy Inverse Reinforcement Learning (ME-IRL) baseline.

Key Insights and Ideas

  • Divide and Conquer is Key: Separating feature learning from reward learning significantly reduces sample complexity and improves generalization. This is because it allows for more targeted human input.
  • Feature Traces are Effective: This novel human input method is intuitive for users and provides structured information that allows for efficient learning of complex, non-linear features.
  • Structure Matters: Explicitly learning features provides a beneficial structure that helps disambiguate reward functions, leading to better performance than end-to-end learning from raw states.
  • Online Adaptation is Powerful: The online FERL approach allows robots to dynamically learn missing features during interaction, making them more robust to incomplete initial feature sets.
  • Non-Expert Users Can Teach Features: The user study demonstrates that even non-experts can effectively teach features using the proposed method, suggesting broader applicability beyond expert system designers.

Key Results

  • Feature Learning Quality: FERL can learn high-quality features that closely resemble ground truth, even with a limited number of feature traces. The quality of learned features improves with more traces.
  • Online FERL Performance: FERL significantly outperforms ME-IRL in reward accuracy and generalization, especially when one feature is missing. It achieves lower MSE to the ground truth reward and assigns higher likelihood to optimal trajectories.
  • Offline FERL Performance: FERL also outperforms ME-IRL in offline reward learning, particularly for more complex rewards involving multiple features. This advantage is more pronounced when features are of higher quality.
  • User Study: Non-expert users were able to teach features with significant signal, achieving MSE values much closer to expert performance than random initialization. Users found the feature teaching protocol intuitive.

Limitations and Future Work

  • Spurious Correlations: High-dimensional state spaces can lead to spurious correlations, affecting feature learning quality, especially with limited data. The paper proposes a subspace selection heuristic but acknowledges that more robust methods (e.g., causal learning) are needed.
  • Noisy Features: When features are learned from noisy data (e.g., user data in simulation), their quality can degrade, potentially diminishing the benefits of the FERL approach, especially in offline reward learning with multiple features.
  • User Study Environment: The user study was conducted in simulation due to COVID-19. Future work should investigate performance with physical robots and a wider range of users.
  • Abstract Features: Teaching highly abstract or discontinuous features might still be challenging.
  • Active Learning: Developing active learning strategies to autonomously select informative feature traces could further reduce human effort.
  • Confidence Estimation: Integrating confidence estimation to determine whether to accept or reject learned features is a promising direction.
  • Higher Dimensional Spaces: Extending FERL to even higher dimensional state spaces, like images, requires further investigation.

Summary updated

Abstract

Reward learning enables robots to learn adaptable behaviors from human input. Traditional methods model the reward as a linear function of hand-crafted features, but that requires specifying all the relevant features a priori, which is impossible for real-world tasks. To get around this issue, recent deep Inverse Reinforcement Learning (IRL) methods learn rewards directly from the raw state but this is challenging because the robot has to implicitly learn the features that are important and how to combine them, simultaneously. Instead, we propose a divide and conquer approach: focus human input specifically on learning the features separately, and only then learn how to combine them into a reward. We introduce a novel type of human input for teaching features and an algorithm that utilizes it to learn complex features from the raw state space. The robot can then learn how to combine them into a reward using demonstrations, corrections, or other reward learning frameworks. We demonstrate our method in settings where all features have to be learned from scratch, as well as where some of the features are known. By first focusing human input specifically on the feature(s), our method decreases sample complexity and improves generalization of the learned reward over a deepIRL baseline. We show this in experiments with a physical 7DOF robot manipulator, as well as in a user study conducted in a simulated environment.
BibTeX
@article{bobu2022inducing,
  author = {Andreea Bobu and Marius Wiggert and Claire J. Tomlin and Anca D. Dragan},
  title = {{Inducing Structure in Reward Learning by Learning Features}},
  journal = {Int. J. Robotics Res.},
  year = {2022},
  doi = {10.1177/02783649221078031},
}
Figure