TL;DR
This paper introduces a new way for AI agents to learn from human feedback in reinforcement learning (RL). Instead of asking humans to compare two options (like "which video is better?"), it asks them to rate a single option on a scale (like "how good is this video?"). This "rating-based" approach is shown to be more efficient and less frustrating for humans, leading to comparable or better learning performance for the AI.
What Problem Does This Paper Address?
Deep Reinforcement Learning (RL) has achieved remarkable success in complex tasks, but a major hurdle is the need for a well-defined reward function. Manually crafting these functions is difficult, time-consuming, and prone to unintended consequences like "reward hacking." While existing methods use human feedback, they often rely on pairwise comparisons (Preference-Based RL or PbRL) or demonstrations. PbRL, while more efficient than demonstrations, can be sample-inefficient because each comparison only provides a single bit of information and doesn't give a global sense of quality. Humans might also struggle to consistently pick the "better" option when both are poor, or when options are very similar. This paper addresses the need for a more intuitive and efficient way to leverage human feedback for RL, specifically by using absolute ratings of individual trajectories.
Methods
The core of this paper is a novel Rating-based Reinforcement Learning (RbRL) approach. Unlike existing methods that rely on relative preferences between sample pairs, RbRL directly uses human ratings of individual trajectories.
Here's a breakdown of the key components:
- Human Ratings: Humans are presented with individual trajectories (sequences of states and actions) and assign a discrete rating from a predefined set of classes (e.g., "good," "bad," or a more granular scale like "very good," "good," "neutral," "bad," "very bad"). The number of rating classes, denoted by n, is flexible.
- Reward Modeling: The system learns a reward function that predicts state-action rewards. It then estimates a normalized discounted return for each trajectory segment, denoted as $\tilde{R}(\sigma)$, which is mapped to a value between 0 and 1.
- Novel Multi-Class Loss Function: A new multi-class cross-entropy loss function is introduced. This loss function models the probability $Q_\sigma(i)$ that a human assigns a specific rating class i to a trajectory segment $\sigma$. This probability is based on the normalized predicted return $\tilde{R}(\sigma)$ and learned boundaries between rating classes.
- Modeling Human Rating Probabilities: The paper proposes a method to model $Q_\sigma(i)$ using a function that depends on the normalized return $\tilde{R}(\sigma)$ and predefined boundaries ($R_i$) for each rating class. This function ensures that the probability of belonging to a class is maximized when the predicted return falls within that class's range.
- Modeling Boundaries: The boundaries between rating classes ($R_i$) are determined by matching the distribution of predicted returns to the distribution of human-assigned ratings in the training data. This ensures that the learned model reflects the proportions of ratings observed from humans.
Datasets
The paper uses the following for evaluation:
- Synthetic Ratings: Generated for individual samples based on ground truth return and predefined rating class boundaries. This was done for the Walker and Quadruped environments.
- Real Human Ratings: Collected from human participants in the Cheetah, Swimmer, and Hopper MuJoCo environments. This study involved both expert (authors) and non-expert users.
The paper does not introduce new benchmark datasets but rather uses existing environments and generates synthetic feedback to compare with existing methods.
Key Insights and Ideas
- Absolute Ratings are Powerful: The core idea that using absolute ratings of individual trajectories can be more informative and efficient than pairwise comparisons is a significant insight. It bypasses some of the limitations of PbRL, such as difficulty with poor-quality options or similar samples.
- Human Intuition is Leveraged: The approach aligns well with how humans naturally evaluate things – by assigning a quality score rather than always comparing two things. This makes the feedback process potentially less demanding and more intuitive.
- The "Sweet Spot" for Rating Classes: The experiments suggest that there's an optimal number of rating classes. Too few might oversimplify, while too many can make distinctions difficult for humans, leading to worse performance. For the tested environments, n between 3 and 7 seemed to be beneficial.
- RbRL is Less Demanding: User studies indicated that participants found the rating-based feedback process less demanding, difficult, and frustrating compared to preference-based feedback.
Key Results
- Performance: In synthetic experiments, RbRL with an appropriate number of rating classes (e.g., n=5, 6) outperformed the PbRL baseline in the Walker environment. In the Quadruped environment, RbRL with n=2, 3 also outperformed PbRL.
- Human Study Performance: In human experiments, RbRL performed comparably to or better than PbRL across different tasks (Cheetah, Swimmer, Hopper). RbRL learned quickly, showing fast reward growth. Expert trials showed RbRL consistently outperforming PbRL.
- Efficiency: Participants could provide more ratings per minute (approx. 14.03) than pairwise preferences (approx. 8.7), indicating a significant time saving (62% more time for preferences). This was particularly pronounced in the Cheetah environment.
- User Experience: A user questionnaire revealed that participants found RbRL less demanding, difficult, and frustrating than PbRL.
Limitations and Future Work
- Inconsistent Ratings: A key limitation is the potential for inconsistent human ratings over time due to fatigue, attention span, or changing understanding of the task. Future work could focus on developing mechanisms to quantify user consistency and mitigate its impact.
- Less Refined Reward Function: RbRL, as presented, doesn't distinguish between samples within the same rating class, potentially leading to a less refined reward function compared to methods that can make finer distinctions.
- Data Imbalance: Different rating classes might have vastly different numbers of samples, leading to imbalanced datasets. Addressing this imbalance could further improve performance.
- Integration with PbRL: The authors suggest integrating RbRL for fast initial learning with PbRL for finer refinement, creating a multi-phase learning strategy.
- Boundary Optimization: For synthetic experiments, the performance could be further improved by optimizing the selection of rating class boundaries.
Summary updated