Abstract
A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets and 7 different perspectives.
Introduction
Evaluating Natural Language Generation (NLG) is notoriously difficult. Traditional metrics like BLEU or ROUGE rely on simple n-gram overlaps, which often fail to capture semantic meaning or factual correctness. While recent "neural" metrics like BERTScore use pre-trained embeddings to measure similarity, there remains a fundamental "disconnect": these models are pre-trained to generate text, yet they are often used only as static feature extractors for evaluation. This under-utilizes the model’s parameters—specifically the output prediction layer. BARTScore aims to bridge this gap by using the model's internal likelihood of generating a specific sequence as the quality score itself.
Methods: Evaluation as Generation
The core idea of BARTScore is that a high-quality hypothesis should be easy for a pre-trained sequence-to-sequence (seq2seq) model to generate given a source or reference text. Unlike previous paradigms that focus on matching or regression, BARTScore treats evaluation as a generation task.
The Scoring Function
BARTScore utilizes the BART model, an encoder-decoder architecture designed for conditional text generation. The score is calculated as the weighted log probability of a target sequence ($y$) given a source sequence ($x$): $$ \text{BARTScore} = \sum_{t=1}^{m} w_t \log P(y_t | y_{<t}, x, \theta) $$
Flexibility through Directionality
By swapping the source and target sequences, BARTScore can evaluate different facets of quality:
- Faithfulness ($s \to h$): Probability of the hypothesis given the source text, useful for assessing factuality.
- Precision ($r \to h$): Probability of the hypothesis given the reference.
- Recall ($h \to r$): Probability of the reference given the hypothesis, useful for content coverage.
- F-score ($r \leftrightarrow h$): The average of Precision and Recall.
Enhancements: Prompting and Fine-tuning
BARTScore can be further improved through:
- Prompting: Adding phrases like "in summary" or "to rephrase it" to the input to better align the evaluation task with BART's pre-training.
- Fine-tuning: Training BART on task-specific data (e.g., the CNN/Daily Mail dataset) to bring the model's domain closer to the evaluation domain.
Experimental Results
The authors conducted an extensive meta-evaluation across 16 datasets and 7 perspectives, including informativeness, fluency, and factuality.
Performance Highlights
- State-of-the-Art: BARTScore outperformed top existing metrics (including BERTScore, MoverScore, and PRISM) in 16 of 22 test settings.
- Factuality: On the Rank19 dataset, BARTScore (fine-tuned on CNN/DM) achieved an accuracy of 0.836 in distinguishing factual from non-factual summaries, nearly matching the human baseline of 0.839.
- Machine Translation: On WMT19, BARTScore with fine-tuning and prompting achieved the highest average Kendall’s Tau correlation (0.337), surpassing even supervised metrics like BLEURT in some language pairs.
Discussion and Conclusion
BARTScore demonstrates that a generative model's "surprise" (negative log-likelihood) is a direct proxy for text quality. A key finding is the power of prompts; simply adding a prompt like "such as" to a translation can boost correlation with human judgments by 3%.
While BARTScore is highly effective, it shows a slight bias toward abstractive summaries over purely extractive ones. Future work may involve automated prompt search (e.g., prefix-tuning) to further optimize performance. As generative models continue to improve, metrics built on this framework will naturally become more accurate, creating a virtuous cycle between generation and evaluation.
Summary updated