VeriFastScore: Speeding up long-form factuality evaluation

Rishanth Rajendhran, Amir Zadeh, Matthew Sarte, Chuan Li, Mohit Iyyer

Abstract

Metrics like FACTSCORE and VERISCORE that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VERIFASTSCORE, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives 4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VERIFASTSCORE model demonstrates strong correlation with the original VERISCORE pipeline at both the example level (r = 0.80) and system level (r = 0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VERISCORE. To facilitate future factuality research, we publicly release our VERIFASTSCORE model and synthetic datasets.

Introduction

As Large Language Models (LLMs) generate increasingly complex responses, verifying their factuality has become a significant bottleneck. Current state-of-the-art metrics like FactScore and VeriScore rely on a multi-stage "pipeline" approach. They first decompose a response into dozens of atomic claims, perform individual search queries for each, and then call an LLM repeatedly to verify each claim against the retrieved evidence. This process is computationally expensive and slow, often exceeding 100 seconds per response, which precludes its use in real-time applications or as a reward model in Reinforcement Learning from Human Feedback (RLHF).

Methods: From Pipelines to Single-Pass Evaluation

VeriFastScore addresses these inefficiencies by collapsing the multi-stage pipeline into a single model call. The core innovation lies in an integrated approach to retrieval and evaluation.

Comparison of the multi-stage VeriScore pipeline versus the integrated, single-pass VeriFastScore architecture.

Instead of per-claim retrieval, VeriFastScore uses each sentence in a response as a search query, concatenating the results into a single evidence context (averaging ~4,000 tokens). A fine-tuned Llama 3.1 8B Instruct model then processes the response and this consolidated evidence block in one forward pass. The model is trained to simultaneously identify all verifiable claims and label them as "Supported" or "Unsupported."

The training process involved two stages: initial fine-tuning on high-quality, claim-level evidence, followed by "annealing" on a mixture of claim-level and noisier sentence-level evidence to ensure robustness under real-world search conditions.

Key Results and Insights

The primary advantage of VeriFastScore is its efficiency. It achieves a 6.6x total speedup over VeriScore, with the modeling component alone being 9.9x faster. Despite this speedup, it maintains high alignment with the original metric, showing a 0.94 system-level correlation.

A critical finding is that this task is too complex for standard few-shot prompting, even with state-of-the-art closed models. As shown in the performance comparison, few-shot GPT-4o struggles significantly with the combined task of decomposition and verification against long, noisy contexts.

Performance comparison between few-shot GPT-4o and the fine-tuned VeriFastScore across different evidence granularities.

The results demonstrate that fine-tuning allows the 8B-parameter model to effectively "decontextualize" claims and resolve coreferences internally—tasks that few-shot models fail to perform reliably at this scale of evidence.

Conclusion and Future Work

VeriFastScore provides a practical path toward scalable factuality evaluation. While it significantly reduces latency and cost, it does introduce a trade-off in interpretability, as it provides a list of labels based on a large evidence block rather than specific snippets for each claim. Future research will explore the use of reasoning-first models to generate explicit rationales, potentially bridging the gap between speed and transparency.

Summary updated

Abstract

Metrics like FACTSCORE and VERISCORE that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim.While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios.To address this, we propose VERIFASTSCORE, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search.We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives 4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence.However, our fine-tuned VER-IFASTSCORE model demonstrates strong correlation with the original VERISCORE pipeline at both the example level (r = 0.80) and system level (r = 0.94) while achieving an overall speedup of 6.6 (9.9 excluding evidence retrieval) over VERISCORE.To facilitate future factuality research, we publicly release our VERIFASTSCORE model and synthetic datasets.
BibTeX
@misc{rajendhran2025verifastscore,
  author = {Rishanth Rajendhran and Amir Zadeh and Matthew Sarte and Chuan Li and Mohit Iyyer},
  title = {{VeriFastScore: Speeding up long-form factuality evaluation}},
  year = {2025},
  eprint = {2505.16973},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2505.16973},
}
Figure