SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA

Sher Badshah, Ali Emami, Hassan Sajjad

TL;DR

The paper introduces Search-AuGmented Evaluation (SAGE), a reference-free framework that evaluates the factuality of Large Language Model (LLM) outputs on free-form question answering. By equipping an LLM judge with an iterative agentic loop—generating web queries, summarizing search results, and reflecting on evidence sufficiency—SAGE achieves near-perfect alignment with human evaluators without requiring costly, static human-annotated ground truths.

What Problem Does This Paper Address?

Evaluating free-form Question Answering (QA) is notoriously difficult. Traditional automatic metrics like Exact Match (EM) and F1 score rely on static reference answers, which fail to capture the diverse phrasing of open-ended responses and are expensive to scale.

To bypass this, researchers often use an "LLM-as-a-judge." However, when evaluating objective factuality, an unguided LLM judge relies entirely on its frozen, pre-trained parametric knowledge. This leads to several critical failure modes:

  • Knowledge Staleness: Judges confidently endorse outdated answers.
  • Verbosity Bias: Longer, detailed answers are rated highly even if they contain errors.
  • Self-Enhancement / Over-Acceptance: Unguided judges systematically over-accept incorrect answers and fabricate rationales to justify their decisions.

Methods

SAGE addresses these limitations by transforming the LLM judge into an active, iterative research agent. Instead of relying on static references, SAGE dynamically gathers external web evidence to verify a candidate answer ($y$).

Figure 1: Given the question “What is the tallest building in the world?” and candidate answer “The tallest building in the world is the Burj Khalifa in Dubai,”, SAGE begins with initial query from the question. The query triggers web searches across multiple sources, followed by evidence summarization to extract key insights. The reflection module assesses evidence sufficiency and relevance, triggering query refinement if needed. After N iterations, the judge synthesizes the evidence to provide a final decision with rationale.

The framework coordinates four distinct roles for the LLM judge across a fixed budget of $N$ iterations (default $N=3$):

  1. Query Generation ($Q$): Formulates an initial search query based on the question ($x$). In later rounds, it refines the query based on missing information stored in short-term memory.
  2. Web Search ($S$): Submits the query to a search engine API (e.g., Serper, Brave, or Tavily) to retrieve real-time web snippets.
  3. Evidence Summarization ($\Sigma$): Condenses raw search snippets into a concise, factual summary, filtering out irrelevant noise.
  4. Iterative Reflection ($\Phi$): Analyzes whether the gathered evidence supports, contradicts, or is inconclusive regarding the candidate answer, flagging gaps to guide the next search iteration.

After $N$ rounds, the accumulated history is passed to the final Judge ($J$), which outputs a binary verdict (True/False) along with a natural-language rationale.

Datasets

The authors evaluated SAGE on 300 randomly sampled instances from five diverse free-form QA benchmarks:

  • AmbigQA: Questions with multiple valid interpretations due to inherent ambiguity.
  • HotpotQA: Multi-hop reasoning questions requiring information synthesis.
  • TriviaQA & Natural Questions (NQ-Open): Open-domain queries testing broad factual recall.
  • FreshQA: A dynamic dataset specifically designed to test knowledge of recent, rapidly changing real-world events.

Key Insights and Ideas

  • Fixing the Over-Acceptance Bias: Without search, LLM judges suffer from a massive false-positive rate—they tend to assume candidate answers are correct. SAGE drastically reduces false positives by forcing the judge to ground its decision in external text.
Figure 2: Distribution of true positive (TP), true negative (TN), false positive (FP), and false negative (FN) rates for each LLM judge in both Judge without search and SAGE, averaged across AmbigQA and HotpotQA and candidate models.
  • The Power of Iteration: A single-pass search baseline (searching once and immediately judging) helps, but the full summarize-reflect-refine loop provides a massive accuracy boost. The reflection step acts as a quality gate, explicitly catching when initial search results are off-topic or insufficient.
  • Catching Flawed Reasoning: SAGE doesn't just check the final answer; its iterative nature allows it to flag cases where a model accidentally arrived at the correct answer through completely fabricated or logically inconsistent intermediate reasoning steps.

Key Results

  • High Human Alignment: On AmbigQA and HotpotQA, SAGE achieved substantial to near-perfect agreement with human majority votes. For instance, using GPT-4o as a judge on HotpotQA, Cohen’s $\kappa$ soared from 0.38 (without search) to 0.70 (with SAGE-3).
  • Robustness Across Search Engines: SAGE's performance gains remain consistent regardless of the underlying search API used (Serper, Brave, or Tavily).
  • Highly Cost-Effective: Running SAGE-3 using gpt-4o-mini costs approximately $0.0043 per instance, making it orders of magnitude cheaper and faster than human annotation pipelines.

Limitations and Future Work

  • Context Window Constraints: Because SAGE appends queries, evidence summaries, and reflections to an append-only short-term memory buffer, long multi-round conversations can quickly bloat the context window. The authors suggest exploring a recall-based long-term memory in future iterations.
  • Source Bias: SAGE assumes web search results are accurate. If multiple online sources echo the same misinformation or hallucinated claim, SAGE has no internal mechanism to discount them.
  • Binary Verdicts: SAGE currently outputs a strict True/False decision, making it less suitable for tasks that require partial-correctness grading or nuanced, rubric-based evaluation.

Summary updated

Abstract

As Large Language Models (LLMs) become increasingly used for question-answering (QA), relying on static, pre-annotated references for evaluation poses significant challenges in cost, scalability, and completeness. Meanwhile, using LLMs themselves as evaluators without external grounding remains unreliable for objective tasks, as they systematically over-accept incorrect answers, fabricate supporting rationales, and degrade sharply on questions that fall outside their training data. We propose Search-AuGmented Evaluation (SAGE), a framework to assess LLM outputs without fixed ground-truth answers. Unlike conventional metrics that compare to static references or depend solely on LLM-as-a-judge knowledge, SAGE acts as an agent that actively retrieves and synthesizes external evidence. It iteratively generates web queries, collects information, summarizes findings, and refines subsequent searches through reflection. By reducing dependence on static reference-driven evaluation protocols, SAGE offers a scalable and adaptive alternative for evaluating the factuality of LLMs. Experimental results on multiple free-form QA benchmarks show that SAGE achieves substantial to perfect agreement with human evaluations.
BibTeX
@inproceedings{badshah2026sage,
  author = {Sher Badshah and Ali Emami and Hassan Sajjad},
  title = {{SAGE: A Search-AuGmented Evaluation of Large Language Models on Free-Form QA}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2026},
  doi = {10.18653/v1/2026.acl-long.66},
}
Figure