Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks

Tianze Xu, Yanzhao Zheng, Pengrui Lu, Lyumanshan Ye, Yong Wu, Zhentao Zhang, Yuanqiang Yu, Chao Ma, Jihuai Zhu, Pengfei Liu, Baohua Dong, Hangcheng Zhu, Ruohui Huang, Gang Yu

TL;DR

The paper introduces Rubrics to Tokens (RTT), a reinforcement learning framework that translates coarse, response-level rubric scores into precise, token-level credit assignments for training Large Language Models (LLMs). By combining a token-level relevance discriminator with a novel intra-sample normalization strategy, RTT overcomes the classic RL pitfalls of reward sparsity and reward ambiguity, significantly boosting an LLM's ability to follow complex, multi-constraint instructions.

What Problem Does This Paper Address?

When aligning LLMs to follow complex instructions (e.g., "Write a story under 500 words starting with 'My Answer' and including the word 'Robot'"), researchers often use Rubric-based Reinforcement Learning. Instead of a simple binary "correct/incorrect" signal, rubrics grade the model on multiple dimensions.

However, existing methods evaluate these rubrics at the response level, leading to two major bottlenecks:

  1. Reward Sparsity: If a model must satisfy all constraints perfectly to get a reward (All-or-Nothing), an untrained model will rarely succeed, leaving it with no helpful gradient signals to learn from.
  2. Reward Ambiguity: If we instead reward the model based on the percentage of constraints met, two completely different responses might get the exact same score for satisfying entirely disjoint subsets of constraints. This confounds the training process because the model doesn't know which specific tokens caused the success or failure.

To fix this, we need a way to map holistic rubric scores down to individual tokens—a challenge known as credit assignment. Furthermore, moving from a simple one-dimensional reward to a three-dimensional reward space (sample, token, and criteria) introduces a new challenge: how do we normalize rewards across these dimensions without introducing bias?

Methods

The authors propose the Rubrics to Tokens (RTT) framework, which consists of three core components:

  1. Token-Level Relevance Discriminator ($D_\phi$): This model predicts which specific tokens in a response are directly responsible for satisfying or violating a given constraint. For instance, if a constraint forbids mathematical formulas, the discriminator flags only the tokens making up the formula.
  2. RTT-GRPO: The authors adapt Group Relative Policy Optimization (GRPO) to handle dense, token-level rewards. The final optimization objective optimizes a combined advantage: $$\hat{A}_{sum}^{(i,t)} = \alpha \hat{A}_{res}^{(i,t)} + \beta \hat{A}_{tok}^{(i,t)}$$ where $\hat{A}_{res}$ is the standard response-level advantage and $\hat{A}_{tok}$ is the newly introduced token-level advantage.

  3. Intra-Sample Token Group Normalization: When calculating token advantages across multiple constraints and generated responses (the "Group Partitioning Problem"), a naive approach would normalize across all tokens of all generated samples combined. However, the authors prove this introduces a severe length bias, where longer responses dominate the variance and suppress learning signals from shorter ones. RTT solves this by normalizing token rewards within each individual response independently before aggregating across constraints.

Architecture of the RTT framework showing response-level and token-level advantage computation pipelines.

Datasets

  • Discriminator Training: The token-level discriminator was trained on a dataset of 62,714 samples constructed from the Tulu-3 SFT corpus and HiR-16K. Negative samples (failed constraint attempts) were engineered using two clever strategies: Minimal Modification (rewriting a good response to violate one rule) and Constraint Omission (withholding a rule during generation).
  • Evaluation Benchmarks:
  • In-Domain (Instruction Following): IFEval, IFBench, MulDimIF, and AdvancedIF.
  • Out-of-Domain (Generalization): MATH-500, GPQA, and MMLU-Pro.

Key Insights and Ideas

  • The Group Partitioning Problem: Expanding the reward space from 1D (outcome-level) to 3D (sample, token, rubric) means that where you draw your normalization boundaries matters immensely.
  • Length Bias is a Hidden Killer: Jointly normalizing token rewards across different generated responses systematically penalizes shorter, concise answers. Isolating the normalization to an intra-sample scope preserves the true token-level intent grounding.
  • Mitigating RL Instability: Standard constraint-rate RL often suffers from severe training instability, characterized by sudden drops in rollout accuracy and spikes in policy entropy. Providing dense, token-level feedback acts as a powerful regularizer, keeping training exceptionally stable.
Figure 2: Reward structure under standard GRPO and RTT-GRPO. In GRPO (left), each of G sampled completions receives one outcome-level reward, advantages are normalized along the sample axis only. The RTT-GRPO (right) assigns rewards over three dimensions, motivating two group normalization strategies, inter-sample and intra-sample group normalization.

Key Results

  • State-of-the-Art Instruction Following: RTT consistently outperformed all baselines (SFT, DPO, and standard RL) across Qwen3-4B, Qwen2.5-7B, and Llama3.2-3B models. Compared to the strongest RL baselines, RTT yielded average gains of 2.50% in instruction-level accuracy and 1.64% in rubric-level accuracy.
  • Preserved Generalization: Even though RTT trains strictly on instruction-following data, it maintained or improved performance on out-of-domain reasoning tasks like MATH-500 and GPQA, proving it doesn't degrade the model's core capabilities.
  • Minimal Computational Overhead: Because the token-level discriminator is forward-only during RL training (requiring no gradient updates or optimizer states), it only adds about 8.5% additional computational overhead compared to standard RL pipelines.

Limitations and Future Work

While Intra-sample Token Group Normalization effectively solves the length-bias problem, the authors note they have only scratched the surface of the 3D reward space. They only compared two representative partitioning strategies (inter-sample vs. intra-sample), leaving more complex multi-reward partitioning combinations unexplored.

For future work, the authors suggest extending this token-level credit assignment paradigm to turn-level credit assignment in multi-step agent training (e.g., tool use and environment interactions), where identifying which specific action led to success or failure remains a critical bottleneck.

Summary updated

Abstract

Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.
BibTeX
@inproceedings{xu2026rubrics,
  author = {Tianze Xu and Yanzhao Zheng and Pengrui Lu and Lyumanshan Ye and Yong Wu and Zhentao Zhang and Yuanqiang Yu and Chao Ma and Jihuai Zhu and Pengfei Liu and Baohua Dong and Hangcheng Zhu and Ruohui Huang and Gang Yu},
  title = {{Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks}},
  booktitle = {COLM},
  year = {2026},
  url = {https://openreview.net/forum?id=QHqWI2cBDr},
}
Figure