Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores

Shukai Liu, Chenming Wu, Ying Li, Liangjun Zhang

TL;DR

This paper introduces a novel interactive reinforcement learning (IRL) method that uses human-provided scores for entire robot trajectories instead of pairwise preferences. This approach significantly improves feedback efficiency, allowing agents to learn complex tasks with less human input by leveraging the richer information contained in scores.

What Problem Does This Paper Address?

Deep Reinforcement Learning (DRL) has shown great promise in robotics, but creating accurate reward functions for complex tasks is challenging. Sparse rewards, while simpler, lead to slow learning. Inverse Reinforcement Learning (IRL) infers rewards from expert demonstrations, but often requires optimal or numerous demonstrations. Interactive IRL methods, which use human feedback during training, aim to alleviate this. However, existing methods like preference-based learning (e.g., PEBBLE, T-REX) often require a large volume of pairwise comparisons between trajectories, which can be tedious and inefficient for humans. This paper addresses the bottleneck of human feedback in interactive IRL by proposing a more efficient scoring-based approach.

Methods

The core of the proposed method lies in using scalar scores for entire trajectories instead of pairwise preferences. The system consists of two main components:

  1. Policy Learning: An off-policy DRL algorithm (SAC in this case) learns the agent's behavior policy. The rewards for this policy are generated by a learned reward network.
  2. Reward Learning: A separate neural network learns the reward function. This network is trained using scores provided by a human teacher on complete agent trajectories.

Key novelties include:

  • Score-based Feedback: Instead of asking "Is trajectory A better than trajectory B?", the teacher provides a score (e.g., 0-10) for trajectory A. The authors argue this yields more information per feedback instance.
  • Adaptive Learning Scheme: To handle imperfect or noisy human scores, they introduce an adaptive learning approach. This involves:
  • Adaptive Network Optimization: The reward network is trained using a learning-to-rank objective, specifically by minimizing cross-entropy loss between predicted and true preferences derived from scores. Crucially, they use a label smoothing technique where the smoothing strength is adaptive to the score difference between trajectory pairs. This means larger score differences lead to more confident (closer to 0 or 1) soft labels, while smaller differences result in softer labels, making the learning more robust to noise.
  • Adaptive Trajectory Sampling: To efficiently select which trajectories to ask for scores, they employ a priority-based sampling strategy. This method prioritizes trajectories that are more informative for learning the reward function, either by random selection or by a probability that increases with the trajectory's score. This is an improvement over entropy-based sampling which might not effectively mine useful trajectories for reward learning.

Datasets

The paper evaluates the proposed method on several continuous control tasks:

  • Mujoco Simulator: Ant and HalfCheetah for locomotion tasks.
  • OpenAI Gym: Not explicitly listed as a dataset, but used in conjunction with Mujoco.
  • Metaworld Environment: PushButton and SweepInto for robotic manipulation tasks.

A "scripted teacher" was used to generate ground truth scores for evaluation purposes by linearly mapping episode returns to a scoring range.

Key Insights and Ideas

  • Scores are More Efficient than Preferences: The fundamental insight is that a single score for a full trajectory provides richer information than a binary preference between two partial trajectories. This directly leads to needing fewer feedback instances.
  • Robustness to Imperfect Scores is Crucial: Human feedback is inherently noisy. The adaptive label smoothing and adaptive trajectory sampling are key to making the system robust to these imperfections, allowing for effective learning even with suboptimal scores.
  • Learning Reward and Policy Simultaneously: The tight coupling between the reward learning and policy learning components allows for continuous refinement of both, leading to efficient convergence.
  • Customized Behavior Inference: The method can infer user intent and learn customized behaviors by adapting to different scoring preferences, as demonstrated in the PushButton task.

Key Results

  • Improved Feedback Efficiency: The proposed method achieves comparable or better performance than PEBBLE (a state-of-the-art preference-based method) using significantly fewer feedback instances (e.g., hundreds of scores vs. thousands of preferences).
  • Near-Optimal Performance: The method successfully trains robotic locomotion and manipulation policies to near-optimal levels in sparse reward environments.
  • Robustness to Noise: The adaptive reward update method demonstrates strong robustness to scoring errors, outperforming standard label smoothing and hard updates.
  • Effective Sampling: Priority-based sampling significantly outperforms entropy-based and uniform sampling for selecting trajectories for reward updates.
  • Real Human Experiments: The method was validated with real human users, showing that good policies could be trained with approximately 300 scores.

Limitations and Future Work

  • Scoring Scale: While the paper uses a 0-10 scoring range, the learned reward function's scale might differ from the ground truth. However, they note this difference did not affect policy learning performance.
  • Computational Cost: Learning a reward function from scratch can initially require more training time compared to using a pre-defined true reward.
  • Future Work: The paper suggests further exploration of adaptive learning schemes and sampling strategies. The ability to infer user intent for customized behaviors is also a promising avenue for future research.

Summary updated

Abstract

Interactive reinforcement learning has shown promise in learning complex robotic tasks. However, the process can be human-intensive due to the requirement of a large amount of interactive feedback. This paper presents a new method that uses scores provided by humans instead of pairwise preferences to improve the feedback efficiency of interactive reinforcement learning. Our key insight is that scores can yield significantly more data than pairwise preferences. Specifically, we require a teacher to interactively score the full trajectories of an agent to train a behavioral policy in a sparse reward environment. To avoid unstable scores given by humans negatively impacting the training process, we propose an adaptive learning scheme. This enables the learning paradigm to be insensitive to imperfect or unreliable scores. We extensively evaluate our method for robotic locomotion and manipulation tasks. The results show that the proposed method can efficiently learn near-optimal policies by adaptive learning from scores while requiring less feedback compared to pairwise preference learning methods. The source codes are publicly available at https://github.com/SSKKai/Interactive-Scoring-IRL.
BibTeX
@inproceedings{liu2023boosting,
  author = {Shukai Liu and Chenming Wu and Ying Li and Liangjun Zhang},
  title = {{Boosting Feedback Efficiency of Interactive Reinforcement Learning by Adaptive Learning from Scores}},
  booktitle = {IEEE/RSJ International Conference on Intelligent Robots and Systems},
  year = {2023},
  url = {https://openreview.net/forum?id=-eiv8azH4X8},
}
Figure