Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment

Dake Zhang, Mark D. Smucker, Charles L. A. Clarke

TL;DR

The paper introduces a reusable evaluation framework and an LLM-based "AutoJudge" for assistive Retrieval-Augmented Generation (RAG) systems designed to help readers assess news trustworthiness. By leveraging expert-crafted, importance-weighted rubrics, the automated judge closely mirrors human evaluation rankings (achieving a Kendall’s $\tau$ up to 0.872), providing a scalable, open-source benchmark for future misinformation-fighting AI assistants.

What Problem Does This Paper Address?

Traditional automated fact-checking treats misinformation as a simple pipeline: identify a claim, retrieve evidence, and predict a binary true/false label. However, real-world news is rarely that black-and-white. Misleading articles often rely on selective omissions, missing context, or questionable sourcing rather than outright lies.

To combat this, researchers are building assistive RAG systems that encourage "lateral reading"—a technique used by professional fact-checkers where they quickly leave a page to investigate the publisher, author, and broader context.

The challenge lies in evaluation. Human assessment of long-form, multi-source reports is the gold standard but is incredibly expensive and static; you cannot easily score a new system once the human judges have gone home. Meanwhile, standard automated metrics like BLEU or ROUGE fail to capture whether a generated report actually covers critical context or introduces subtle contradictions.

Methods

As the organizers of the TREC 2025 DRAGUN (Detection, Retrieval, and Augmented Generation for Understanding News) track, the authors developed a "rubric-first" evaluation methodology consisting of two tasks:

  1. Task 1 (Question Generation): Systems must generate 10 ranked investigative questions a reader should ask about a target news article.
  2. Task 2 (Report Generation): Systems must generate a 250-word well-attributed report grounded in a corpus to help readers evaluate the article's trustworthiness.

To evaluate these tasks sustainably, the authors designed AutoJudge, an automated evaluation pipeline powered by an open-weight LLM (gpt-oss-120b) with few-shot prompting.

Instead of comparing system text to a single gold-standard reference text, AutoJudge uses human-crafted rubrics as a checklist. For question generation, it evaluates the semantic similarity between system-generated questions and rubric questions. For report generation, it reads the system's report and determines whether it supports, partially supports, contradicts, or has no connection (none) to each specific short answer in the rubric.

Datasets

The authors have released a comprehensive package of resources, which includes:

  • The Topic Set: 30 controversial news articles from 28 different media sources, carefully selected to span a wide spectrum of political biases (left, right, neutral, pro-science, and conspiracy-pseudoscience) published between 2019 and 2021.
  • Expert-Authored Rubrics: 30 importance-weighted rubrics created by TREC assessors via open-web research, containing a total of 236 questions and 551 expected short answers with supporting reference URLs.
  • Target Retrieval Corpus: The evaluation is grounded in the MS MARCO V2.1 Segmented Corpus, which contains approximately 114 million text segments.
  • Human Judgments: Over 12,000 judged question pairs and 15,000 judged answer-report pairs from the official TREC track to serve as a validation baseline.

Key Insights and Ideas

  • Avoiding the "Pool-Nuggets" Pitfall: Traditional information retrieval evaluations extract information "nuggets" from what participating systems happen to retrieve. The DRAGUN track avoids this by having humans build rubrics from the open web before looking at system submissions. This ensures systems are rewarded for finding critical context they might have otherwise completely missed.
  • Substantial Headroom for AI Assistants: By plotting individual system performance against an "all runs combined" ceiling, the authors highlighted a massive gap between current single-system capabilities and the theoretical upper bound of what can be extracted from the corpus.
Per-topic rubric-weighted scores showing the headroom between the best single run and the pooled upper bound.
  • Contradictions are Rare, Omissions are Common: The evaluation revealed that current LLM-based RAG systems rarely generate explicit contradictions to the facts. Instead, their main failure mode is omission—failing to retrieve or synthesize the most critical investigative angles within tight length constraints.

Key Results

The primary validation of the paper centers on how well the automated AutoJudge correlates with official TREC human assessments:

  • Task 1 (Question Generation): AutoJudge achieved a run-level rank correlation of Kendall’s $\tau = 0.678$ and an 82.1% raw label agreement with humans.
  • Task 2 (Report Generation): The alignment was exceptionally strong, achieving a Kendall’s $\tau = 0.872$ and an 86.7% raw label agreement.
Correlation between human and LLM-based automatic assessments at the run level.

These high correlation scores demonstrate that the AutoJudge can successfully stand in for human assessors to rank future iterations of RAG systems reliably and automatically.

Limitations and Future Work

  • Post-Hoc Filtering Artifacts: The track guidelines prohibited compound questions (e.g., "Who is X and when did Y happen?"). The authors used an LLM to filter these out post-hoc. However, because human judging happened before filtering, a system could be penalized if its best-matching question was filtered out, even if a valid non-compound alternative existed lower in its submission list.
  • Narrow Evaluation Dimensions: The current AutoJudge focuses strictly on content coverage (factual support vs. contradiction). It does not evaluate other critical dimensions of RAG performance, such as citation faithfulness (checking if the cited document actually supports the claim made by the LLM) or overall text readability.
  • Corpus Mismatch: Because human rubrics were built using the open web, some expert-identified facts may not exist within the static MS MARCO corpus, creating an artificial performance ceiling for systems restricted to that corpus.

Summary updated

Abstract

Many readers today struggle to assess the trustworthiness of online news because reliable reporting coexists with misinformation. The TREC 2025 DRAGUN (Detection, Retrieval, and Augmented Generation for Understanding News) Track provided a venue for researchers to develop and evaluate assistive RAG systems that support readers' news trustworthiness assessment by producing reader-oriented, well-attributed reports. As the organizers of the DRAGUN track, we describe the resources that we have newly developed to allow for the reuse of the track's tasks. The track had two tasks: (Task 1) Question Generation, producing 10 ranked investigative questions; and (Task 2, the main task) Report Generation, producing a 250-word report grounded in the MS MARCO V2.1 Segmented Corpus. As part of the track's evaluation, we had TREC assessors create importance-weighted rubrics of questions with expected short answers for 30 different news articles. These rubrics represent the information that assessors believe is important for readers to assess an article's trustworthiness. The assessors then used their rubrics to manually judge the participating teams' submitted runs. To make these tasks and their rubrics reusable, we have created an automated process to judge runs not part of the original assessing. We show that our AutoJudge ranks existing runs well compared to the TREC human-assessed evaluation (Kendall's $τ= 0.678$ for Task 1 and $τ= 0.872$ for Task 2). These resources enable both the evaluation of RAG systems for assistive news trustworthiness assessment and, with the human evaluation as a benchmark, research on improving automated RAG evaluation.
BibTeX
@inproceedings{zhang2026resources,
  author = {Dake Zhang and Mark D. Smucker and Charles L. A. Clarke},
  title = {{Resources for Automated Evaluation of Assistive RAG Systems that Help Readers with News Trustworthiness Assessment}},
  booktitle = {International ACM SIGIR Conference on Research and Development in Information Retrieval},
  year = {2026},
  doi = {10.1145/3805712.3808624},
}
Figure