TL;DR
This paper introduces a framework that separates reward specification from reward computation for open-ended LLM post-training. By generating reusable, prompt-specific rubrics and executable code verifiers offline, it creates a transparent, hybrid reward system that significantly improves both offline evaluation and online reinforcement learning without requiring human annotations or reference answers.
What Problem Does This Paper Address?
Post-training large language models (LLMs) for open-ended tasks—such as creative writing, instruction following, or decision support—relies heavily on reinforcement learning (RL). However, designing effective reward functions for these tasks is notoriously difficult.
Standard paradigms like Reinforcement Learning from Human Feedback (RLHF) rely on opaque, scalar reward models where the prompt-specific criteria remain implicit. On the flip side, Reinforcement Learning with Verifiable Rewards (RLVR) provides clean, explicit signals but is restricted to domains with deterministic ground truths (like math or coding) and fails on open-ended tasks. Generic LLM-as-a-judge approaches can score open-ended text, but their criteria are often inconsistent and reconstructed on the fly for every single rollout.
The authors address this "prompt-level reward specification problem" by making success conditions explicit, structured, and reusable before responses are even generated.
Methods
The proposed framework splits the process into two stages: Offline Reward Specification Construction and Online Reward Computation.
1. Offline Construction
- Task-Adaptive Rubric Construction: A lightweight classifier categorizes the prompt (e.g., writing, reasoning, planning). A general-purpose LLM then uses a shared template combined with task-specific plug-ins to generate atomic, weighted criteria ($\mathcal{R}_x$) tailored to that specific prompt.
- Hard-Constraint Checker Construction: In parallel, explicit surface-level constraints (like word counts, formatting rules, or forbidden strings) are extracted and compiled into executable Python code checkers ($\mathcal{C}_x$).
2. Online Computation
When a response is generated during training or evaluation, it passes through three parallel lanes:
- Rubric-Based Scoring ($s_r$): An LLM evaluates the response against each atomic criterion independently using a ternary scale (yes, part, or no).
- Global Scoring ($s_g$): An independent LLM provides a holistic score (0-10) to capture overall coherence and interactions between criteria.
- Code-Based Scoring ($s_c$): The compiled Python code executes in a sandbox to calculate a deterministic constraint pass rate.
These three normalized signals are aggregated into a single unified hybrid reward, with the global score's weight linearly decaying over the course of RL training.
Datasets
Training Prompts
A mixture of approximately 13K prompts was curated for online RL training:
- VERINSTRUCT (5K prompts): Focused on explicit instruction following and verifiable constraints.
- DeepWriting-20K (5K prompts): Focused on open-ended writing quality, coherence, and style.
- Synthetic Decision-Support Prompts (3K prompts): Newly synthesized by the authors to cover scenario-based tasks requiring risk assessment, trade-off analysis, and planning under uncertainty.
Evaluation Benchmarks
- Offline Reward Evaluation: RewardBench v2 and RM-Bench.
- Online RL Evaluation: IFEval, IFBench, Arena-Hard-v2.0, Creative Writing v3, and WritingBench.
- Reasoning Capabilities: GSM8K, GPQA Diamond, and AIME 2024 (used to monitor potential capability regression).
Key Insights and Ideas
- Decoupling Specification from Computation: Defining what success looks like before rollout generation ensures that all candidate responses to the same prompt are judged against a fixed baseline, preventing the judge from shifting goalposts dynamically.
- Complementarity of Local vs. Global Signals: Decomposed rubrics excel at tracking fine-grained requirements, but they can miss holistic properties like flow or conflicting trade-offs. Combining them with an independent global score creates a more balanced, dense reward signal.
- Code as a Targeted Stabilizer: Executable code checkers do not replace semantic judgment, but they dramatically reduce reward failures on strict formatting or length constraints, stabilizing the RL advantage signal.
- Preserving Reasoning: Training on open-ended tasks using this hybrid reward does not cause catastrophic forgetting or performance drops on hard reasoning benchmarks like math and science.
Key Results
- Offline Performance: The hybrid reward approach consistently improves the accuracy of underlying LLM judges. When applied to a Qwen3.5-35B-A3B backbone, it achieved an overall score of 85.1 on RewardBench v2, outperforming proprietary judges like GPT-4.1 and Gemini-2.5-pro (see Table 1).
- Online RL Gains: Across multiple policy backbones (including DeepSeek-R1-Distill-Qwen-7B and GLM-4.7-Flash), online RL using the hybrid reward yielded substantial average score improvements (ranging from +4.7 to +8.7 points) across open-ended benchmarks like IFEval and Creative Writing (see Table 2).
- Ablation Proof: Component ablations confirmed that removing any single component (rubric, global, or code) resulted in a performance drop, validating the hybrid design (see Table 3).
Limitations and Future Work
- Dependence on Strong LLMs: The framework relies heavily on advanced LLMs (like GPT-5 or large open-weight models) to extract high-quality rubrics and constraints offline. Errors in the initial rubric extraction can propagate downstream.
- Fixed Aggregation Rules: The scores from the three lanes are combined using a fixed, heuristic aggregation formula and a simple linear decay schedule, which has not been dynamically optimized.
- Code Sandbox Risks: Executing LLM-generated Python code introduces security risks, requiring strict sandboxing, restricted permissions, and auditing before deployment.
- Future Directions: The authors suggest exploring alternative rubric extraction strategies, finer criterion granularities, and automated rubric refinement procedures.
Summary updated