Abstract
Metrics like FactScore and VeriScore that evaluate long-form factuality operate by decomposing an input response into atomic claims and then individually verifying each claim. While effective and interpretable, these methods incur numerous LLM calls and can take upwards of 100 seconds to evaluate a single response, limiting their practicality in large-scale evaluation and training scenarios. To address this, we propose VeriFastScore, which leverages synthetic data to fine-tune Llama3.1 8B for simultaneously extracting and verifying all verifiable claims within a given text based on evidence from Google Search. We show that this task cannot be solved via few-shot prompting with closed LLMs due to its complexity: the model receives ~4K tokens of evidence on average and needs to concurrently decompose claims, judge their verifiability, and verify them against noisy evidence. However, our fine-tuned VeriFastScore model demonstrates strong correlation with the original VeriScore pipeline at both the example level (r=0.80) and system level (r=0.94) while achieving an overall speedup of 6.6x (9.9x excluding evidence retrieval) over VeriScore. To facilitate future factuality research, we publicly release our VeriFastScore model and synthetic datasets.
Introduction
As Large Language Models (LLMs) generate increasingly complex long-form content, verifying the factuality of these responses becomes a significant bottleneck. Current "gold standard" metrics like FactScore and VeriScore employ a "decompose-then-verify" approach: breaking responses into atomic claims, searching for evidence for each, and using an LLM to judge each claim individually. This multi-stage process is computationally expensive and slow, often exceeding 100 seconds per response, which precludes its use in real-time applications or as a reward model for Reinforcement Learning from Human Feedback (RLHF).
Methods
VeriFastScore collapses the traditional multi-stage pipeline into a single model pass. Instead of iterative searching, it treats each sentence in the response as a search query and aggregates the top results into a consolidated evidence context of approximately 4,000 tokens.
The system utilizes a fine-tuned Llama 3.1 8B Instruct model trained via synthetic data distillation. The model was trained on 9,000 prompt-response pairs labeled by the original VeriScore pipeline. This fine-tuning allows the model to handle the complex task of simultaneous claim extraction, de-contextualization (e.g., resolving coreferences), and verification against noisy evidence—a task where few-shot prompting of closed models like GPT-4o typically fails.
Results
VeriFastScore achieves a 6.6x overall speedup and a 9.9x speedup in the modeling phase compared to VeriScore, while also reducing retrieval costs by 39%. Despite the increased speed, the model maintains high fidelity to the original metric, with a Pearson correlation of $r=0.80$ at the example level.
At the system level, VeriFastScore produces model rankings that are nearly identical to VeriScore ($r=0.94$), as shown in the comparison of various LLMs. Human evaluation confirmed a verification accuracy of 91.5%. Furthermore, the model's ability to see the entire response at once allows for better de-contextualization of claims compared to sentence-focused pipelines.
Conclusion
VeriFastScore provides a practical solution for large-scale factuality evaluation by significantly reducing latency without sacrificing accuracy. While current limitations include potential positional bias in extremely long texts and a reliance on English-language data, the efficiency gains make it a viable candidate for integration into training loops and large-scale auditing.
Summary updated