Introduction
Reinforcement Learning (RL) performance is heavily dependent on reward design. While "sparse" rewards—where an agent only receives feedback upon task completion—are theoretically ideal, they often make learning computationally intractable in complex environments. To bridge this gap, researchers employ "shaping rewards" to provide intermediate guidance. However, manual reward shaping is a labor-intensive "dark art," prone to unintended agent behaviors where the agent maximizes the shaping signal without solving the primary task.
Existing automated methods often rely on "naive" selection, training multiple policies in parallel for various candidate rewards. This approach is prohibitively expensive. This paper introduces ORSO (Online Reward Selection and Policy Optimization), a framework that treats reward design as an online model selection problem, significantly reducing the computational overhead of finding optimal shaping functions.
Methodology: The ORSO Framework
ORSO decouples the reward design process into two distinct phases: generation and selection.
1. Reward Generation
Rather than manual engineering, ORSO leverages Large Language Models (LLMs), specifically GPT-4, to generate a diverse pool of candidate reward functions in Python. To ensure robustness, a rejection sampling step filters out any code that fails to execute within the simulation environment.
2. Online Reward Selection
The core innovation of ORSO is treating the selection of these candidates as a multi-armed bandit problem. Instead of training $K$ policies to completion, ORSO adaptively allocates training iterations to the most promising rewards based on their performance relative to the "true" task reward.
To handle the non-stationary nature of this problem—where a reward's utility changes as the agent's policy evolves—the authors utilize Doubling Data-Driven Regret Balancing (D3RB). This algorithm provides theoretical guarantees for converging on the optimal reward function by balancing the "regret" (performance lost by selecting sub-optimal rewards) across the candidate pool.
Experimental Evaluation
The researchers evaluated ORSO using the Isaac Gym high-performance physics engine across six continuous control tasks:
- Basic Control: Cartpole and BallBalance.
- Locomotion: Ant and Humanoid, requiring shaping for gait stability.
- Complex Manipulation: AllegroHand and ShadowHand, involving high-dimensional multi-fingered robotic manipulation.
Key Results and Insights
ORSO demonstrates significant improvements in both efficiency and final policy performance.
- Computational Efficiency: ORSO achieves human-level performance with up to an 8x reduction in GPU hours compared to naive parallel selection methods.
- Performance Gains: The framework identified reward functions that outperformed prior automated methods, such as EUREKA, by an average of 56%.
- Competitive with Experts: In several benchmarks, the rewards discovered by ORSO produced policies that matched or exceeded the performance of those designed by human domain experts.
The success of ORSO highlights that while LLMs are proficient at generating reward components (e.g., joint torque penalties), they lack the internal physics model to predict optimal weighting. ORSO serves as the necessary empirical filter, allowing the system to "fail fast" on poor candidates and double down on winners.
Limitations and Future Directions
Despite its success, ORSO currently requires a predefined "ground truth" task reward for evaluation, which may be difficult to define for qualitative goals like "aesthetic movement." Future research may integrate Vision-Language Models (VLMs) to provide qualitative feedback. Additionally, while data-efficient, the current selection process is largely sequential; future iterations could explore more sophisticated parallelization across large-scale compute clusters.
Summary updated