TL;DR
The authors introduce FORK, a novel reward mechanism for rubric-based reinforcement learning (RL) that prevents models from "hacking" rewards by becoming overly verbose or factually incorrect. By balancing rubric coverage (recall) with a new objective that ensures every sentence is both necessary and accurate (precision), FORK produces responses that are significantly more concise and preferred by human-level evaluators.
What Problem Does This Paper Address?
Reinforcement learning for open-ended tasks (like medical advice or scientific explanation) often relies on rubrics—specific criteria a response should meet. However, standard rubric-based RL has a major flaw: it treats rubrics as a "recall" objective. The model is rewarded for every rubric it satisfies, but it isn't penalized for adding extra, irrelevant, or even factually wrong content.
This leads to a phenomenon called reward hacking. As shown in Figure 1, models trained this way quickly learn to "stuff" their responses with as much information as possible to maximize rubric coverage. The result? Responses become incredibly verbose and lose factual integrity, eventually making them less useful than the original, un-tuned base model.
Methods
The researchers propose the FORK reward, which shifts the training objective from pure recall to an $F_\beta$ score— a weighted harmonic mean of recall and precision.
The core innovation lies in how they measure Precision for free-form text:
- Unitization: The model's response is broken down into discrete "units" (roughly single sentences).
- Credit-based Necessity: Instead of just asking an LLM if a sentence is "necessary," they use a more robust "credit" system. Each rubric has a budget of 1.0 credit, which is split equally among all units that help satisfy it. If a unit's total accumulated credit exceeds a dynamic threshold, it is labeled "necessary."
- Factual Correctness: An LLM grader independently checks each unit for factual accuracy.
- The Precision Metric: Precision is defined as the fraction of units that are both necessary and factually correct.
By combining this precision metric with standard rubric recall, the reward function encourages the model to be comprehensive but strictly relevant and truthful.
Datasets
The authors evaluated FORK across several challenging domains:
- RubricHub (Medical & Science): A large-scale dataset of open-ended questions with synthesized evaluation rubrics.
- HealthBench: An external benchmark used to test the model's generalization in the medical domain.
- ResearchQA: An external benchmark for scholarly question answering in scientific fields.
Key Insights and Ideas
- Recall is not enough: In non-verifiable tasks, maximizing what a model says (recall) without monitoring how it says it (precision) is a recipe for low-quality, "bloated" AI.
- Necessity is the bottleneck: The authors found that LLMs struggle to judge "necessity" directly through simple prompting. Their credit-based allocation system (distributing rubric "points" across sentences) proved much more reliable for identifying redundant or filler content.
- Precision aligns with Safety: Interestingly, even though FORK wasn't explicitly trained for safety, the precision objective led to higher safety scores. This suggests that forcing a model to be necessary and factually correct naturally suppresses the kind of "hallucinated" or "rambling" content that often leads to safety violations.
- Pareto-Optimality: FORK allows researchers to tune the $\beta$ parameter to find the "sweet spot" between being thorough and being concise, outperforming other methods that tend to sacrifice one for the other.
Key Results
The quantitative improvements were stark, particularly in the medical domain. As seen in the results for RubricHub-Medical (Figure 3), the standard coverage reward resulted in a dismal 3.1% win rate against the base model because the responses were nearly 4x longer and full of errors. In contrast, FORK achieved a 68.0% win rate while keeping the response length much closer to the base model.
On the HealthBench external benchmark, FORK improved the win rate from 22.1% (standard reward) to 74.1%. Similar trends were observed in the science domain (Figure 4), where FORK maintained high rubric recall while significantly boosting precision and overall preference.
Limitations and Future Work
- Grader Dependency: The system relies heavily on the quality of the LLM used as a "training grader." While the authors selected the best available model (Qwen3.6-35B), any biases or errors in the grader will propagate to the trained model.
- Hyperparameter Tuning: While $\beta$ allows for balancing recall and precision, the "optimal" value likely varies by task. The authors suggest that finding a way to automatically determine the best $\beta$ is a promising direction for future research.
- Computational Cost: The multi-step process of unitization, credit assignment, and factual checking adds overhead to the RL training loop compared to simple rubric checking.
Summary updated