TL;DR
Reinforcement learning with verifiable rewards excels in math and coding but struggles in open-ended medical QA due to the lack of cheap, automated outcome verifiers. ConRub-Med addresses this by generating high-quality consensus rubrics from a panel of LLMs, applying a three-state scoring system that penalizes errors, and resolving reward ties with bidirectional pairwise comparisons. This approach achieves state-of-the-art performance across multiple medical benchmarks while curbing reward hacking and preserving general instruction-following capabilities.
What Problem Does This Paper Address?
Reinforcement Learning from Verifiable Rewards (RLVR) has driven massive breakthroughs in domains like mathematics and software engineering. These fields offer cheap, objective, and repeatable checks (like compiler outputs or exact answer matches) to evaluate thousands of model rollouts without human intervention.
Open-ended medical question answering (QA) enjoys no such luxury. Medical answers are rarely just "right" or "wrong"—they can be partially correct but incomplete, or mostly accurate but contain a single, highly dangerous clinical error. While human-expert rubrics provide strong clinical grounding, involving physicians to score every training trajectory is prohibitively expensive and unscalable.
While model-generated rubrics offer a scalable alternative, they often suffer from quality gaps compared to expert standards. Furthermore, when optimizing models using Group Relative Policy Optimization (GRPO), responses frequently receive identical scalar rewards as the policy improves. These exact ties result in zero gradient advantages, effectively stalling the model's learning process.
Methods
ConRub-Med introduces a robust pipeline designed to preserve meaningful clinical distinctions all the way from rubric construction to policy optimization:
- Consensus Rubric Construction: To eliminate the biases or hallucinations of any single model, three distinct LLMs (GPT-5 Mini, Gemini 2.5 Pro, and Claude Sonnet 4.5) independently propose atomic evaluation criteria for a given prompt. A separate reviewer model (DeepSeek-V4-Pro) filters and clusters these proposals, retaining only the criteria that receive unanimous semantic support from all three generators.
- Three-State Criterion Scoring: Instead of binary (yes/no) feedback, a judge model evaluates responses using three states: correct (+1), missing (0), or wrong (-1). Assigning negative credit to explicit errors prevents the model from "reward hacking"—a common phenomenon where models simply write longer, fluff-filled answers to maximize the chance of triggering positive criteria.
- Pairwise Sequence Advantages for Ties: When all responses within a GRPO rollout group yield identical scalar rewards, vanilla GRPO provides zero optimization signal. ConRub-Med breaks these ties by invoking a pairwise judge (Qwen3-32B) to compare response pairs bidirectionally (evaluating both A vs B and B vs A to eliminate position bias). If the preferences are order-consistent, they are written directly into the token-level advantages, restoring a gradient signal without altering the base scalar rewards or introducing a separate preference loss.
Datasets
The authors constructed a training dataset consisting of 5,166 unique prompts balanced across several provenances: 70% RaR-Medicine, 20% RubricHub Medical, 5% RubricHub Chat, and 5% RubricHub Instruction-Following. This dataset yielded 59,378 consensus instances (comprising 49,046 prompt-specific content criteria and 10,332 global controls).
Evaluation was conducted across nine diverse benchmarks:
- Medical QA: HealthBench-Hard, MedXpertQA-Text, DiagnosisArena-MCQ, MedMCQA, PubMedQA, and MMLU-Medical.
- Generalization & Capability Retention: WritingBench, GPQA-Diamond, and IFEval-Loose.
Key Insights and Ideas
- Penalizing Errors Curbs Verbosity: Binary scoring systems naturally incentivize models to generate longer answers because longer text has a higher probability of mentioning a required keyword or concept. By explicitly penalizing wrong information (-1), the three-state scoring system successfully reduced response lengths and minimized text truncation during training.
- Ties are a Goldmine for Generalization: As reinforcement learning progresses, models get better, causing exact reward ties to skyrocket (reaching ~7.7% of all rollout groups by epoch 3). Resolving these ties via pairwise comparisons provided a fine-grained preference signal that prevented the model from over-specializing. While it only slightly altered medical averages, it massively boosted general instruction-following performance (IFEval).
- Cross-Model Consensus Mimics Experts: In a blinded study, medical experts rated the criteria generated by the ConRub-Med consensus pipeline as significantly more clinically relevant (99.3%) than panels generated by any single model alone (which hovered around 86%).
Key Results
- Benchmark Leadership: Among open-weight models, ConRub-Med ranked first on 6 out of 9 benchmarks, achieving the highest medical macro-average (51.76) and generalization macro-average (74.27).
- HealthBench-Hard Superiority: ConRub-Med achieved a score of 38.98 ± 1.04 on HealthBench-Hard. This comfortably outperforms the strong InfiMed-ORBIT baseline (37.30), despite InfiMed requiring more than five times the amount of training samples (28,000 vs. ConRub-Med's 5,166).
- Signal Recovery: Pairwise advantages successfully restored training directions in tied groups, resulting in a significantly higher pre-clipping gradient norm throughout all three training epochs compared to standard three-state optimization without pairwise intervention.
Limitations and Future Work
- Consensus Bias: If all three generator models share a common medical misconception, the consensus pipeline will inadvertently reinforce it. Furthermore, strict unanimous filtering may discard highly nuanced or specialized criteria caught by only one or two models.
- Compute Overhead: Generating rubrics requires querying multiple commercial closed-source APIs, and employing an online 32B model as a judge during RL training imposes substantial computational costs.
- Strict Tie Dependency: The pairwise advantage mechanism is currently restricted to exact scalar ties. The authors note that extending this approach to "near-ties" will require developing a calibrated margin that preserves meaningful scalar reward differences.
Summary updated