Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL

Hyun Ryu, Jong-Heon Kim, Yiqing Xie, Seungone Kim, Carolyn Rose, Eunho YangarXiv 2026 · arXiv

TL;DR

The authors introduce FORK, a novel reward mechanism for rubric-based reinforcement learning (RL) that prevents models from "hacking" rewards by becoming overly verbose or factually incorrect. By balancing rubric coverage (recall) with a new objective that ensures every sentence is both necessary and accurate (precision), FORK produces responses that are significantly more concise and preferred by human-level evaluators.

What Problem Does This Paper Address?

Reinforcement learning for open-ended tasks (like medical advice or scientific explanation) often relies on rubrics—specific criteria a response should meet. However, standard rubric-based RL has a major flaw: it treats rubrics as a "recall" objective. The model is rewarded for every rubric it satisfies, but it isn't penalized for adding extra, irrelevant, or even factually wrong content.

This leads to a phenomenon called reward hacking. As shown in Figure 1, models trained this way quickly learn to "stuff" their responses with as much information as possible to maximize rubric coverage. The result? Responses become incredibly verbose and lose factual integrity, eventually making them less useful than the original, un-tuned base model.

Figure 1: Reward hacking in rubric-based RL and our mitigation method. With the standard coverage reward, recall (i.e., weighted coverage of rubrics) quickly saturates, while precision and win rate against a base model continually decreases and responses become more verbose. With a precision objective included in the reward (FORK), the model reaches similar recall while gradually increasing precision and win rate and generating less verbose responses. Here, we train Qwen2.5- 7B-Instruct on RubricHub-Medical and evaluate it on an in-domain test set.

Methods

The researchers propose the FORK reward, which shifts the training objective from pure recall to an $F_\beta$ score— a weighted harmonic mean of recall and precision.

The core innovation lies in how they measure Precision for free-form text:

  1. Unitization: The model's response is broken down into discrete "units" (roughly single sentences).
  2. Credit-based Necessity: Instead of just asking an LLM if a sentence is "necessary," they use a more robust "credit" system. Each rubric has a budget of 1.0 credit, which is split equally among all units that help satisfy it. If a unit's total accumulated credit exceeds a dynamic threshold, it is labeled "necessary."
  3. Factual Correctness: An LLM grader independently checks each unit for factual accuracy.
  4. The Precision Metric: Precision is defined as the fraction of units that are both necessary and factually correct.

By combining this precision metric with standard rubric recall, the reward function encourages the model to be comprehensive but strictly relevant and truthful.

Figure 2: Training with FORK. Given a prompt, a policy model samples a response, which is then divided into units. An LLM grader matches rubrics to units to assign each unit a necessity credit and determines a credit threshold for unit necessity. In parallel, the grader assesses the factual correctness of all units. Precision is defined as the fraction of necessary and factually correct units, and recall follows the standard definition of weighted rubric coverage. We update the policy model using their weighted harmonic mean as a reward.

Datasets

The authors evaluated FORK across several challenging domains:

  • RubricHub (Medical & Science): A large-scale dataset of open-ended questions with synthesized evaluation rubrics.
  • HealthBench: An external benchmark used to test the model's generalization in the medical domain.
  • ResearchQA: An external benchmark for scholarly question answering in scientific fields.

Key Insights and Ideas

  • Recall is not enough: In non-verifiable tasks, maximizing what a model says (recall) without monitoring how it says it (precision) is a recipe for low-quality, "bloated" AI.
  • Necessity is the bottleneck: The authors found that LLMs struggle to judge "necessity" directly through simple prompting. Their credit-based allocation system (distributing rubric "points" across sentences) proved much more reliable for identifying redundant or filler content.
  • Precision aligns with Safety: Interestingly, even though FORK wasn't explicitly trained for safety, the precision objective led to higher safety scores. This suggests that forcing a model to be necessary and factually correct naturally suppresses the kind of "hallucinated" or "rambling" content that often leads to safety violations.
  • Pareto-Optimality: FORK allows researchers to tune the $\beta$ parameter to find the "sweet spot" between being thorough and being concise, outperforming other methods that tend to sacrifice one for the other.

Key Results

The quantitative improvements were stark, particularly in the medical domain. As seen in the results for RubricHub-Medical (Figure 3), the standard coverage reward resulted in a dismal 3.1% win rate against the base model because the responses were nearly 4x longer and full of errors. In contrast, FORK achieved a 68.0% win rate while keeping the response length much closer to the base model.

Table 1: Main results on medical domain. We conduct rubric-based and rubric-free evaluations of our method and other rubric-based RL baselines on the RubricHub-Medical test set and Health- Bench. Rubric-based evaluation measures macro rubric recall, macro precision, and the macro F1 score. Precision indicates the fraction of necessary and factually correct response units. Rubric-free evaluation measures win rate against the base policy model and average response length. We use GPT-5.4 as a reference grader for automating these evaluations.

On the HealthBench external benchmark, FORK improved the win rate from 22.1% (standard reward) to 74.1%. Similar trends were observed in the science domain (Figure 4), where FORK maintained high rubric recall while significantly boosting precision and overall preference.

Table 2: Main results on science domain. We evaluate rubric-based RL methods on the RubricHub- Science test set and ResearchQA as in Table 1.

Limitations and Future Work

  • Grader Dependency: The system relies heavily on the quality of the LLM used as a "training grader." While the authors selected the best available model (Qwen3.6-35B), any biases or errors in the grader will propagate to the trained model.
  • Hyperparameter Tuning: While $\beta$ allows for balancing recall and precision, the "optimal" value likely varies by task. The authors suggest that finding a way to automatically determine the best $\beta$ is a promising direction for future research.
  • Computational Cost: The multi-step process of unitization, credit assignment, and factual checking adds overhead to the RL training loop compared to simple rubric checking.

Summary updated

Abstract

Rubric-based rewards enable effective reinforcement learning on non-verifiable tasks. However, because these rewards are typically defined as a weighted measure of rubric coverage, models are trained primarily to maximize a recall objective. As a result, RL-trained models produce responses that are more complete but also more verbose and contain more unnecessary or factually incorrect content, leading them to be less preferred than responses from the base model. To address this issue, we present the FORK reward, which jointly optimizes recall and a novel precision objective, which requires each unit of a model response to be both necessary and factually correct, as measured by rubric-driven necessity credits and a factual-correctness grader, respectively. Our experimental results show that FORK outperforms existing rubric-based RL methods and achieves Pareto-optimal performance that balances rubric coverage and overall response quality. Compared to the original recall-based reward, FORK substantially improves precision and reduces response length while achieving similar rubric coverage, which leads to significantly higher win rate compared to the base policy (e.g., from 10.6% to 68.0% on RubricHub-Medical and from 22.1% to 74.1% on HealthBench).
BibTeX
@misc{ryu2026eat,
  author = {Hyun Ryu and Jong-Heon Kim and Yiqing Xie and Seungone Kim and Carolyn Rose and Eunho Yang},
  title = {{Eat Your Precision: Precision Objective Mitigates Reward Hacking in Rubric-based RL}},
  year = {2026},
  eprint = {2610.33512},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.33512},
}
Figure