TL;DR
The paper introduces a reusable evaluation framework and an LLM-based "AutoJudge" for assistive Retrieval-Augmented Generation (RAG) systems designed to help readers assess news trustworthiness. By leveraging expert-crafted, importance-weighted rubrics, the automated judge closely mirrors human evaluation rankings (achieving a Kendall’s $\tau$ up to 0.872), providing a scalable, open-source benchmark for future misinformation-fighting AI assistants.
What Problem Does This Paper Address?
Traditional automated fact-checking treats misinformation as a simple pipeline: identify a claim, retrieve evidence, and predict a binary true/false label. However, real-world news is rarely that black-and-white. Misleading articles often rely on selective omissions, missing context, or questionable sourcing rather than outright lies.
To combat this, researchers are building assistive RAG systems that encourage "lateral reading"—a technique used by professional fact-checkers where they quickly leave a page to investigate the publisher, author, and broader context.
The challenge lies in evaluation. Human assessment of long-form, multi-source reports is the gold standard but is incredibly expensive and static; you cannot easily score a new system once the human judges have gone home. Meanwhile, standard automated metrics like BLEU or ROUGE fail to capture whether a generated report actually covers critical context or introduces subtle contradictions.
Methods
As the organizers of the TREC 2025 DRAGUN (Detection, Retrieval, and Augmented Generation for Understanding News) track, the authors developed a "rubric-first" evaluation methodology consisting of two tasks:
- Task 1 (Question Generation): Systems must generate 10 ranked investigative questions a reader should ask about a target news article.
- Task 2 (Report Generation): Systems must generate a 250-word well-attributed report grounded in a corpus to help readers evaluate the article's trustworthiness.
To evaluate these tasks sustainably, the authors designed AutoJudge, an automated evaluation pipeline powered by an open-weight LLM (gpt-oss-120b) with few-shot prompting.
Instead of comparing system text to a single gold-standard reference text, AutoJudge uses human-crafted rubrics as a checklist. For question generation, it evaluates the semantic similarity between system-generated questions and rubric questions. For report generation, it reads the system's report and determines whether it supports, partially supports, contradicts, or has no connection (none) to each specific short answer in the rubric.
Datasets
The authors have released a comprehensive package of resources, which includes:
- The Topic Set: 30 controversial news articles from 28 different media sources, carefully selected to span a wide spectrum of political biases (left, right, neutral, pro-science, and conspiracy-pseudoscience) published between 2019 and 2021.
- Expert-Authored Rubrics: 30 importance-weighted rubrics created by TREC assessors via open-web research, containing a total of 236 questions and 551 expected short answers with supporting reference URLs.
- Target Retrieval Corpus: The evaluation is grounded in the MS MARCO V2.1 Segmented Corpus, which contains approximately 114 million text segments.
- Human Judgments: Over 12,000 judged question pairs and 15,000 judged answer-report pairs from the official TREC track to serve as a validation baseline.
Key Insights and Ideas
- Avoiding the "Pool-Nuggets" Pitfall: Traditional information retrieval evaluations extract information "nuggets" from what participating systems happen to retrieve. The DRAGUN track avoids this by having humans build rubrics from the open web before looking at system submissions. This ensures systems are rewarded for finding critical context they might have otherwise completely missed.
- Substantial Headroom for AI Assistants: By plotting individual system performance against an "all runs combined" ceiling, the authors highlighted a massive gap between current single-system capabilities and the theoretical upper bound of what can be extracted from the corpus.
- Contradictions are Rare, Omissions are Common: The evaluation revealed that current LLM-based RAG systems rarely generate explicit contradictions to the facts. Instead, their main failure mode is omission—failing to retrieve or synthesize the most critical investigative angles within tight length constraints.
Key Results
The primary validation of the paper centers on how well the automated AutoJudge correlates with official TREC human assessments:
- Task 1 (Question Generation): AutoJudge achieved a run-level rank correlation of Kendall’s $\tau = 0.678$ and an 82.1% raw label agreement with humans.
- Task 2 (Report Generation): The alignment was exceptionally strong, achieving a Kendall’s $\tau = 0.872$ and an 86.7% raw label agreement.
These high correlation scores demonstrate that the AutoJudge can successfully stand in for human assessors to rank future iterations of RAG systems reliably and automatically.
Limitations and Future Work
- Post-Hoc Filtering Artifacts: The track guidelines prohibited compound questions (e.g., "Who is X and when did Y happen?"). The authors used an LLM to filter these out post-hoc. However, because human judging happened before filtering, a system could be penalized if its best-matching question was filtered out, even if a valid non-compound alternative existed lower in its submission list.
- Narrow Evaluation Dimensions: The current AutoJudge focuses strictly on content coverage (factual support vs. contradiction). It does not evaluate other critical dimensions of RAG performance, such as citation faithfulness (checking if the cited document actually supports the claim made by the LLM) or overall text readability.
- Corpus Mismatch: Because human rubrics were built using the open web, some expert-identified facts may not exist within the static MS MARCO corpus, creating an artificial performance ceiling for systems restricted to that corpus.
Summary updated