ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization

Chen Bo Calvin Zhang, Zhang-Wei Hong, Aldo Pacchiano, Pulkit Agrawal

Introduction

Reinforcement Learning (RL) performance is heavily dependent on reward design. While "sparse" rewards—where an agent only receives feedback upon task completion—are theoretically ideal, they often make learning computationally intractable in complex environments. To bridge this gap, researchers employ "shaping rewards" to provide intermediate guidance. However, manual reward shaping is a labor-intensive "dark art," prone to unintended agent behaviors where the agent maximizes the shaping signal without solving the primary task.

Existing automated methods often rely on "naive" selection, training multiple policies in parallel for various candidate rewards. This approach is prohibitively expensive. This paper introduces ORSO (Online Reward Selection and Policy Optimization), a framework that treats reward design as an online model selection problem, significantly reducing the computational overhead of finding optimal shaping functions.

Methodology: The ORSO Framework

ORSO decouples the reward design process into two distinct phases: generation and selection.

1. Reward Generation

Rather than manual engineering, ORSO leverages Large Language Models (LLMs), specifically GPT-4, to generate a diverse pool of candidate reward functions in Python. To ensure robustness, a rejection sampling step filters out any code that fails to execute within the simulation environment.

2. Online Reward Selection

The core innovation of ORSO is treating the selection of these candidates as a multi-armed bandit problem. Instead of training $K$ policies to completion, ORSO adaptively allocates training iterations to the most promising rewards based on their performance relative to the "true" task reward.

Reward selection strategies illustrating the trade-off between exploration and exploitation.

To handle the non-stationary nature of this problem—where a reward's utility changes as the agent's policy evolves—the authors utilize Doubling Data-Driven Regret Balancing (D3RB). This algorithm provides theoretical guarantees for converging on the optimal reward function by balancing the "regret" (performance lost by selecting sub-optimal rewards) across the candidate pool.

Experimental Evaluation

The researchers evaluated ORSO using the Isaac Gym high-performance physics engine across six continuous control tasks:

  • Basic Control: Cartpole and BallBalance.
  • Locomotion: Ant and Humanoid, requiring shaping for gait stability.
  • Complex Manipulation: AllegroHand and ShadowHand, involving high-dimensional multi-fingered robotic manipulation.

Key Results and Insights

ORSO demonstrates significant improvements in both efficiency and final policy performance.

  • Computational Efficiency: ORSO achieves human-level performance with up to an 8x reduction in GPU hours compared to naive parallel selection methods.
  • Performance Gains: The framework identified reward functions that outperformed prior automated methods, such as EUREKA, by an average of 56%.
  • Competitive with Experts: In several benchmarks, the rewards discovered by ORSO produced policies that matched or exceeded the performance of those designed by human domain experts.
ORSO performance across six continuous control tasks compared to human and no-design baselines.

The success of ORSO highlights that while LLMs are proficient at generating reward components (e.g., joint torque penalties), they lack the internal physics model to predict optimal weighting. ORSO serves as the necessary empirical filter, allowing the system to "fail fast" on poor candidates and double down on winners.

Limitations and Future Directions

Despite its success, ORSO currently requires a predefined "ground truth" task reward for evaluation, which may be difficult to define for qualitative goals like "aesthetic movement." Future research may integrate Vision-Language Models (VLMs) to provide qualitative feedback. Additionally, while data-efficient, the current selection process is largely sequential; future iterations could explore more sophisticated parallelization across large-scale compute clusters.

Summary updated

BibTeX
@inproceedings{zhang2025orso,
  author = {Chen Bo Calvin Zhang and Zhang-Wei Hong and Aldo Pacchiano and Pulkit Agrawal},
  title = {{ORSO: Accelerating Reward Design via Online Reward Selection and Policy Optimization}},
  booktitle = {International Conference on Learning Representations},
  year = {2025},
  url = {https://openreview.net/forum?id=0uRc3CfJIQ},
}
Figure