An Efficient Rubric-based Generative Verifier for Search-augmented LLMs

Linyue Ma, Yilong Xu, Xiang Long, Zhi ZhengSubmission to ICLR 2026 · Rejected

TL;DR

Training search-augmented LLMs with reinforcement learning usually relies either on brittle exact-match rules or computationally heavy, non-verifiable pairwise reward models. This paper introduces the “nugget-as-rubric” paradigm, which unifies short- and long-form evaluation by treating atomic factual units as structured criteria. The authors build an automated query-rewriting pipeline to discover these rubrics and train Search-Gen-V, a 4B-parameter generative verifier that matches the accuracy of 200B+ teacher models at a fraction of the inference cost.

What Problem Does This Paper Address?

Reinforcement Learning with Verifiable Rewards (RLVR) has dramatically boosted model reasoning in domains with clear programmatic checks, such as math and code. However, applying RL to search-augmented generation is fundamentally trickier:

  1. Rule-based rewards (e.g., Exact Match or token F1) work only for simple, short-form answers and frequently yield false negatives due to minor phrasing differences. They cannot evaluate multi-paragraph, report-style outputs.
  2. Generative reward models (often pairwise preference rankers) handle long-form generation more flexibly, but they are prone to reward hacking, lack verifiability, and introduce severe throughput bottlenecks during RL rollouts.
  3. Manual rubric annotation does not scale to dynamic web corpora and suffers from pool bias—evaluators only judge top-ranked documents, leaving undiscovered nuggets unrewarded.

Methods

The authors introduce an end-to-end framework to construct and verify objective rewards across both short- and long-form tasks:

[ Question ] ──▶ [ Automated Rubric Pipeline ] ──▶ [ Nugget Rubrics (Vital / Okay) ]
                                                            │
[ Model Answer ] ──▶ [ Paragraph Segmentation ] ───────────┼──▶ [ Search-Gen-V (4B) ] ──▶ Verifiable Reward
  • Nugget-as-Rubric Paradigm: Ground truth is decomposed into atomic, 10–20 word declarative statements (“nuggets”). Short-form answers map to a single nugget, while long-form answers map to an ensemble of weighted nuggets (categorized as vital = 1.0 or okay = 0.5).
  • Automated Rubric Construction: To avoid pool bias without manual labeling, an LLM-guided pipeline constructs an alternating search tree of queries and retrieved passages via rule-guided query rewriting (synonyms, constraints, entity changes). It applies temporal consistency checks, deduplicates overlapping claims, and weights the resulting nuggets.
  • Search-Gen-V (4B): A compact verifier initialized from Qwen3-4B-Instruct-2507. Long answers are broken into paragraph blocks, and the verifier assigns ternary labels (support, partially support, not support) across rubric batches. Block labels are max-pooled to compute an aggregate answer reward.
  • Two-Stage Training:
  • SFT: Distills reasoning and verification decisions from Gemini-2.5-Flash across 10 diverse output serialization formats (JSON, YAML, Markdown, etc.).
  • RL via DAPO: Uses Decoupled Clip and Dynamic Sampling Policy Optimization with a composite objective balancing verification accuracy (70%), short chain-of-thought format (20%), output formatting (10%), and an overlength penalty.

Datasets

  • Rubric & Training Data: Constructed using 207 questions from the TREC Deep Learning Track (2021–2023) with MS MARCO V2.1 passages, plus 3,000 queries sampled from Researchy Questions.
  • Validation: 84 questions from the TREC RAG24 test set.
  • Short-Form Evaluation: 1,000 validation instances each from HotpotQA and TriviaQA.
  • Long-Form Evaluation: 50 multi-faceted questions (yielding 119 filtered long-form answers) from DeepResearch Bench, leveraging live web search (DuckDuckGo + Jina Reader).
  • Preference Benchmark: Factuality subset of RewardBench 2.

Key Insights and Ideas

  • Granular, binary/ternary entailment provides hack-resistance: By framing generative reward modeling as fine-grained fact entailment against pre-mined atomic rubrics rather than holistic pairwise preference ranking, the reward remains verifiable and interpretable.
  • Tree-based query rewriting breaks pool bias: Iteratively mutating query constraints and entities uncovers peripheral, relevant web passages that standard top-$k$ retrieval misses.
  • Small verifiers can rival massive LLMs in structured verification: While 4B base models struggle with rubric verification out-of-the-box (F1 ~0.42–0.58), distilling both reasoning steps and labels allows Search-Gen-V-4B to approach the performance of a 235B-parameter model.
  • Remedying false negatives in rule-based verification: When combined with Exact Match on short-form benchmarks, Search-Gen-V rescues valid paraphrases that Exact Match falsely penalizes, increasing overall precision and recall.

Key Results

  • TREC RAG24 Validation: Search-Gen-V-4B (SFT+RL) achieved an Average F1 of 0.72 (0.70 rubric-level F1, 0.73 sample-level F1), outperforming its base model (0.58 F1) and approaching Qwen3-235B-A22B (0.74 F1).
  • DeepResearch Bench: Evaluated against Gemini-2.5-Pro's comprehensive metric, Search-Gen-V scores reached a Pearson correlation of $r = 0.70$ ($p = 6 \times 10^{-19}$), achieving a verification F1 of 0.57 (compared to 0.42 for the base 4B model and 0.61 for 235B).
  • Short-Form Remediation: On 585 HotpotQA answers misjudged by Exact Match, Search-Gen-V attained 0.63 F1 (approaching 235B's 0.69). Hybrid verification (EM + Search-Gen-V-4B) pushed HotpotQA F1 to 0.94 and TriviaQA F1 to 0.90.
  • RewardBench 2 Factuality: Scored 85.8%, performing competitively with LMUnit-Qwen2.5-72B (87.2%) and surpassing Skywork-Reward-V2-8B (84.6%).

Limitations and Future Work

  • High Rubric Generation Latency: Generating atomic rubrics via iterative LLM search tree expansion is slow, requiring roughly 1 to 2 hours per question.
  • Offline Evaluation Only: The verifier was evaluated on static outputs; the authors have not yet measured throughput, stability, or policy gains inside a live, closed-loop RL training run.
  • Dataset Scope: Evaluations focused on a single representative dataset per task category; broader domain generalization remains to be explored.

Summary updated

Abstract

Search augmentation empowers Large Language Models with retrieval capabilities to overcome the limitations imposed by static parameters. Recently, Reinforcement Learning leverages tailored reward signals as a viable technique to enhance LLMs performing tasks involving search. However, existing reward modeling for search-augmented LLMs faces several limitations. Rule-based rewards, such as Exact Match, are verifiable but fragile to variations in expression and cannot be applied to long-form workloads. In contrast, generative rewards improve robustness, but designing verifiable and stable rewards for long-form workloads in dynamic corpora remains challenging and also incurs high computational costs. In this paper, we propose a unified and verifiable paradigm, ``nugget-as-rubric", which treats atomic information points as structured evaluation criteria for different search-augmentation workloads. Short-form tasks correspond to a single rubric, whereas long-form tasks expand to multiple rubrics aligned with the question’s information needs. To support long-form settings, we design an automatic rubric construction pipeline based on query rewriting, which can automatically retrieve passages relevant to each question and extract rubrics from them, both from static corpora and from dynamic online web content. Furthermore, we introduce \textbf{Search-Gen-V}, a 4B-parameter efficient generative verifier under our proposed verifiable paradigm, which is trained via the idea of distillation and a two-stage strategy. Experimental results show that Search-Gen-V achieves strong verification accuracy across different workloads, making it a scalable, robust, and efficient verifiable reward constructor for search-augmented LLMs.
BibTeX
@misc{ma2025efficient,
  author = {Linyue Ma and Yilong Xu and Xiang Long and Zhi Zheng},
  title = {{An Efficient Rubric-based Generative Verifier for Search-augmented LLMs}},
  year = {2025},
  eprint = {2510.14660},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.14660},
}
Figure