Prompt-Level Reward Specifications for Open-Ended Post-Training

Zijun Weng, Xiaohui Hu, Shuangyong Song, Yongxiang Li, Kaidong Yu, Xuanjing HuangarXiv 2026 · arXiv

TL;DR

This paper introduces a framework that separates reward specification from reward computation for open-ended LLM post-training. By generating reusable, prompt-specific rubrics and executable code verifiers offline, it creates a transparent, hybrid reward system that significantly improves both offline evaluation and online reinforcement learning without requiring human annotations or reference answers.

What Problem Does This Paper Address?

Post-training large language models (LLMs) for open-ended tasks—such as creative writing, instruction following, or decision support—relies heavily on reinforcement learning (RL). However, designing effective reward functions for these tasks is notoriously difficult.

Standard paradigms like Reinforcement Learning from Human Feedback (RLHF) rely on opaque, scalar reward models where the prompt-specific criteria remain implicit. On the flip side, Reinforcement Learning with Verifiable Rewards (RLVR) provides clean, explicit signals but is restricted to domains with deterministic ground truths (like math or coding) and fails on open-ended tasks. Generic LLM-as-a-judge approaches can score open-ended text, but their criteria are often inconsistent and reconstructed on the fly for every single rollout.

The authors address this "prompt-level reward specification problem" by making success conditions explicit, structured, and reusable before responses are even generated.

Methods

The proposed framework splits the process into two stages: Offline Reward Specification Construction and Online Reward Computation.

Figure 1: Left: offline reward specification construction builds reusable reward artifacts from prompts alone, including prompt-specific rubrics Rx and executable hard-constraint checkers Cx. Right: online reward computation combines rubric-based, global, and code-based scoring to produce a unified reward for evaluation and training.

1. Offline Construction

  • Task-Adaptive Rubric Construction: A lightweight classifier categorizes the prompt (e.g., writing, reasoning, planning). A general-purpose LLM then uses a shared template combined with task-specific plug-ins to generate atomic, weighted criteria ($\mathcal{R}_x$) tailored to that specific prompt.
  • Hard-Constraint Checker Construction: In parallel, explicit surface-level constraints (like word counts, formatting rules, or forbidden strings) are extracted and compiled into executable Python code checkers ($\mathcal{C}_x$).

2. Online Computation

When a response is generated during training or evaluation, it passes through three parallel lanes:

  1. Rubric-Based Scoring ($s_r$): An LLM evaluates the response against each atomic criterion independently using a ternary scale (yes, part, or no).
  2. Global Scoring ($s_g$): An independent LLM provides a holistic score (0-10) to capture overall coherence and interactions between criteria.
  3. Code-Based Scoring ($s_c$): The compiled Python code executes in a sandbox to calculate a deterministic constraint pass rate.

These three normalized signals are aggregated into a single unified hybrid reward, with the global score's weight linearly decaying over the course of RL training.

Datasets

Training Prompts

A mixture of approximately 13K prompts was curated for online RL training:

  • VERINSTRUCT (5K prompts): Focused on explicit instruction following and verifiable constraints.
  • DeepWriting-20K (5K prompts): Focused on open-ended writing quality, coherence, and style.
  • Synthetic Decision-Support Prompts (3K prompts): Newly synthesized by the authors to cover scenario-based tasks requiring risk assessment, trade-off analysis, and planning under uncertainty.

Evaluation Benchmarks

  • Offline Reward Evaluation: RewardBench v2 and RM-Bench.
  • Online RL Evaluation: IFEval, IFBench, Arena-Hard-v2.0, Creative Writing v3, and WritingBench.
  • Reasoning Capabilities: GSM8K, GPQA Diamond, and AIME 2024 (used to monitor potential capability regression).

Key Insights and Ideas

  • Decoupling Specification from Computation: Defining what success looks like before rollout generation ensures that all candidate responses to the same prompt are judged against a fixed baseline, preventing the judge from shifting goalposts dynamically.
  • Complementarity of Local vs. Global Signals: Decomposed rubrics excel at tracking fine-grained requirements, but they can miss holistic properties like flow or conflicting trade-offs. Combining them with an independent global score creates a more balanced, dense reward signal.
  • Code as a Targeted Stabilizer: Executable code checkers do not replace semantic judgment, but they dramatically reduce reward failures on strict formatting or length constraints, stabilizing the RL advantage signal.
  • Preserving Reasoning: Training on open-ended tasks using this hybrid reward does not cause catastrophic forgetting or performance drops on hard reasoning benchmarks like math and science.

Key Results

  • Offline Performance: The hybrid reward approach consistently improves the accuracy of underlying LLM judges. When applied to a Qwen3.5-35B-A3B backbone, it achieved an overall score of 85.1 on RewardBench v2, outperforming proprietary judges like GPT-4.1 and Gemini-2.5-pro (see Table 1).
Table 1: Results on RewardBench v2 and RM-Bench.
  • Online RL Gains: Across multiple policy backbones (including DeepSeek-R1-Distill-Qwen-7B and GLM-4.7-Flash), online RL using the hybrid reward yielded substantial average score improvements (ranging from +4.7 to +8.7 points) across open-ended benchmarks like IFEval and Creative Writing (see Table 2).
Table 2: Experimental results of RL training across multiple policy backbones.
  • Ablation Proof: Component ablations confirmed that removing any single component (rubric, global, or code) resulted in a performance drop, validating the hybrid design (see Table 3).
Table 3: Component ablation of the proposed hybrid reward on RewardBench v2.

Limitations and Future Work

  • Dependence on Strong LLMs: The framework relies heavily on advanced LLMs (like GPT-5 or large open-weight models) to extract high-quality rubrics and constraints offline. Errors in the initial rubric extraction can propagate downstream.
  • Fixed Aggregation Rules: The scores from the three lanes are combined using a fixed, heuristic aggregation formula and a simple linear decay schedule, which has not been dynamically optimized.
  • Code Sandbox Risks: Executing LLM-generated Python code introduces security risks, requiring strict sandboxing, restricted permissions, and auditing before deployment.
  • Future Directions: The authors suggest exploring alternative rubric extraction strategies, finer criterion granularities, and automated rubric refinement procedures.

Summary updated

Abstract

Open-ended post-training benefits from rewards that make prompt-specific success conditions explicit, rather than relying only on post-hoc scalar scores. In instruction following, writing, and decision-support tasks, response quality depends on local requirements, holistic preferences, and explicit constraints, but existing reward methods often leave these criteria implicit or cover only narrowly verifiable cases. We propose a prompt-level reward specification framework that separates reward specification from reward computation. Given only prompts, our framework constructs reusable task-adaptive rubrics and executable hard-constraint checkers offline, making reward criteria explicit before training and reusable across rollouts. At scoring time, artifact-anchored rubric and code scores are combined with an independent global score for residual holistic quality, yielding a normalized hybrid reward over requirement satisfaction, holistic quality, and deterministic constraints. The framework requires no human preference annotations, reference answers, or a separately trained reward model. Experiments show that the resulting reward improves offline RM-style response ranking and supports online reinforcement learning across multiple open-ended benchmarks. Ablations further show that rubrics, global scoring, and executable verification provide complementary supervision.
BibTeX
@misc{weng2026prompt,
  author = {Zijun Weng and Xiaohui Hu and Shuangyong Song and Yongxiang Li and Kaidong Yu and Xuanjing Huang},
  title = {{Prompt-Level Reward Specifications for Open-Ended Post-Training}},
  year = {2026},
  eprint = {2605.29275},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2605.29275},
}
Figure