TL;DR
Training search-augmented LLMs with reinforcement learning usually relies either on brittle exact-match rules or computationally heavy, non-verifiable pairwise reward models. This paper introduces the “nugget-as-rubric” paradigm, which unifies short- and long-form evaluation by treating atomic factual units as structured criteria. The authors build an automated query-rewriting pipeline to discover these rubrics and train Search-Gen-V, a 4B-parameter generative verifier that matches the accuracy of 200B+ teacher models at a fraction of the inference cost.
What Problem Does This Paper Address?
Reinforcement Learning with Verifiable Rewards (RLVR) has dramatically boosted model reasoning in domains with clear programmatic checks, such as math and code. However, applying RL to search-augmented generation is fundamentally trickier:
- Rule-based rewards (e.g., Exact Match or token F1) work only for simple, short-form answers and frequently yield false negatives due to minor phrasing differences. They cannot evaluate multi-paragraph, report-style outputs.
- Generative reward models (often pairwise preference rankers) handle long-form generation more flexibly, but they are prone to reward hacking, lack verifiability, and introduce severe throughput bottlenecks during RL rollouts.
- Manual rubric annotation does not scale to dynamic web corpora and suffers from pool bias—evaluators only judge top-ranked documents, leaving undiscovered nuggets unrewarded.
Methods
The authors introduce an end-to-end framework to construct and verify objective rewards across both short- and long-form tasks:
[ Question ] ──▶ [ Automated Rubric Pipeline ] ──▶ [ Nugget Rubrics (Vital / Okay) ]
│
[ Model Answer ] ──▶ [ Paragraph Segmentation ] ───────────┼──▶ [ Search-Gen-V (4B) ] ──▶ Verifiable Reward
- Nugget-as-Rubric Paradigm: Ground truth is decomposed into atomic, 10–20 word declarative statements (“nuggets”). Short-form answers map to a single nugget, while long-form answers map to an ensemble of weighted nuggets (categorized as vital = 1.0 or okay = 0.5).
- Automated Rubric Construction: To avoid pool bias without manual labeling, an LLM-guided pipeline constructs an alternating search tree of queries and retrieved passages via rule-guided query rewriting (synonyms, constraints, entity changes). It applies temporal consistency checks, deduplicates overlapping claims, and weights the resulting nuggets.
- Search-Gen-V (4B): A compact verifier initialized from
Qwen3-4B-Instruct-2507. Long answers are broken into paragraph blocks, and the verifier assigns ternary labels (support, partially support, not support) across rubric batches. Block labels are max-pooled to compute an aggregate answer reward. - Two-Stage Training:
- SFT: Distills reasoning and verification decisions from
Gemini-2.5-Flashacross 10 diverse output serialization formats (JSON, YAML, Markdown, etc.). - RL via DAPO: Uses Decoupled Clip and Dynamic Sampling Policy Optimization with a composite objective balancing verification accuracy (70%), short chain-of-thought format (20%), output formatting (10%), and an overlength penalty.
Datasets
- Rubric & Training Data: Constructed using 207 questions from the TREC Deep Learning Track (2021–2023) with MS MARCO V2.1 passages, plus 3,000 queries sampled from Researchy Questions.
- Validation: 84 questions from the TREC RAG24 test set.
- Short-Form Evaluation: 1,000 validation instances each from HotpotQA and TriviaQA.
- Long-Form Evaluation: 50 multi-faceted questions (yielding 119 filtered long-form answers) from DeepResearch Bench, leveraging live web search (DuckDuckGo + Jina Reader).
- Preference Benchmark: Factuality subset of RewardBench 2.
Key Insights and Ideas
- Granular, binary/ternary entailment provides hack-resistance: By framing generative reward modeling as fine-grained fact entailment against pre-mined atomic rubrics rather than holistic pairwise preference ranking, the reward remains verifiable and interpretable.
- Tree-based query rewriting breaks pool bias: Iteratively mutating query constraints and entities uncovers peripheral, relevant web passages that standard top-$k$ retrieval misses.
- Small verifiers can rival massive LLMs in structured verification: While 4B base models struggle with rubric verification out-of-the-box (F1 ~0.42–0.58), distilling both reasoning steps and labels allows Search-Gen-V-4B to approach the performance of a 235B-parameter model.
- Remedying false negatives in rule-based verification: When combined with Exact Match on short-form benchmarks, Search-Gen-V rescues valid paraphrases that Exact Match falsely penalizes, increasing overall precision and recall.
Key Results
- TREC RAG24 Validation: Search-Gen-V-4B (SFT+RL) achieved an Average F1 of 0.72 (0.70 rubric-level F1, 0.73 sample-level F1), outperforming its base model (0.58 F1) and approaching
Qwen3-235B-A22B(0.74 F1). - DeepResearch Bench: Evaluated against Gemini-2.5-Pro's comprehensive metric, Search-Gen-V scores reached a Pearson correlation of $r = 0.70$ ($p = 6 \times 10^{-19}$), achieving a verification F1 of 0.57 (compared to 0.42 for the base 4B model and 0.61 for 235B).
- Short-Form Remediation: On 585 HotpotQA answers misjudged by Exact Match, Search-Gen-V attained 0.63 F1 (approaching 235B's 0.69). Hybrid verification (
EM + Search-Gen-V-4B) pushed HotpotQA F1 to 0.94 and TriviaQA F1 to 0.90. - RewardBench 2 Factuality: Scored 85.8%, performing competitively with
LMUnit-Qwen2.5-72B(87.2%) and surpassingSkywork-Reward-V2-8B(84.6%).
Limitations and Future Work
- High Rubric Generation Latency: Generating atomic rubrics via iterative LLM search tree expansion is slow, requiring roughly 1 to 2 hours per question.
- Offline Evaluation Only: The verifier was evaluated on static outputs; the authors have not yet measured throughput, stability, or policy gains inside a live, closed-loop RL training run.
- Dataset Scope: Evaluations focused on a single representative dataset per task category; broader domain generalization remains to be explored.
Summary updated