Introduction
Reinforcement Learning (RL) typically relies on meticulously handcrafted reward functions. However, designing these functions is notoriously difficult and often leads to "reward hacking," where agents exploit flaws in the reward definition rather than learning the intended behavior. To mitigate this, researchers have turned to Learning from Human Feedback (LfHF).
While Inverse Reinforcement Learning (IRL) and Preference-based RL (PbRL) are common paradigms, they suffer from significant limitations. IRL requires expert demonstrations that are often difficult for humans to provide, while PbRL—which asks humans to compare pairs of trajectories—suffers from low information density (one bit per comparison) and high cognitive load. This paper introduces Rating-Based Reinforcement Learning (RbRL), a framework where humans provide absolute scores (e.g., 1–5 stars) on individual behavior segments, offering a more sample-efficient and less mentally taxing alternative.
Methodology
RbRL trains a reward model $\hat{r}$ to predict state-action rewards based on discrete human ratings. The framework incorporates several technical innovations to handle the nuances of human feedback:
- Multi-class Cross-entropy Loss: Unlike the binary classifiers used in PbRL, RbRL utilizes a loss function that treats ratings as distinct class labels.
- Probabilistic Rating Model: To account for human noise, the system models the probability of a trajectory segment belonging to a specific rating class using a distribution centered between learned boundaries.
- Dynamic Boundary Modeling: The algorithm automatically adjusts reward thresholds for each rating class based on the agent's current experiences. This ensures the internal reward scale remains aligned with the human's rating behavior as the agent improves.
Experimental Evaluation
The approach was evaluated using standard continuous control benchmarks in the MuJoCo physics engine, including Walker, Quadruped, Cheetah, and Swimmer tasks. Additionally, a custom "Hopper Backflip" task was used to test the agent's ability to learn complex maneuvers without a predefined standard reward function.
Results and Key Insights
Performance and Efficiency
In synthetic tests, RbRL consistently demonstrated superior performance compared to PbRL baselines. Specifically, in the Walker task, agents trained with 5 or 6 rating classes achieved higher mean rewards and learned faster in the early stages of training due to the richer signal provided by multi-class feedback.
The "Goldilocks" Scale
A critical finding of the study is the impact of the number of rating classes ($n$). The data suggests an optimal range of 3–7 classes. Too few classes (such as binary good/bad) lose valuable nuance, while too many (8+) make it difficult for humans to provide consistent labels, leading to increased noise.
Human-Centric Advantages
A user study involving 20 participants revealed that humans found RbRL significantly more efficient and less frustrating. Participants could provide approximately 14 ratings per minute, compared to only 9 preferences in the same timeframe. Qualitatively, RbRL was rated as less mentally demanding, as absolute ratings provide a "global" context that prevents agents from getting stuck in local optima when all current trajectories are poor.
Conclusion and Future Work
RbRL represents a significant step forward in making human-in-the-loop RL more practical. While it excels at global ranking and early-stage learning, it can struggle with fine-grained "intra-class" nuances (distinguishing between two different 5-star performances). Future research may explore hybrid approaches that use RbRL for rapid initial training followed by PbRL for fine-tuning, as well as mechanisms to better filter noisy human labels resulting from fatigue.
Summary updated