BARTScore: Evaluating Generated Text as Text Generation

Weizhe Yuan, Graham Neubig, Pengfei Liu

TL;DR

BARTScore flips the script on automated text evaluation by treating it as a text generation problem rather than a matching or regression task. By using pre-trained sequence-to-sequence models (like BART) to calculate the probability of generating a hypothesis from a source or reference text, BARTScore provides a highly flexible, unsupervised metric that achieves state-of-the-art correlation with human judgments across multiple NLP tasks and evaluation perspectives.

What Problem Does This Paper Address?

Evaluating the quality of generated text (like machine translation or summarization) is notoriously difficult. While human evaluation is the gold standard, it is expensive and slow. To automate this, the NLP community has recently shifted toward using pre-trained language models as evaluators (e.g., BERTScore, MoverScore).

However, there is a fundamental disconnect in how these metrics work: they use models pre-trained on text generation objectives merely as downstream feature extractors, completely ignoring the decoder and the generation head. Furthermore, most existing metrics are narrow, focusing only on one or two perspectives (like semantic equivalence or fluency) and struggling to evaluate complex dimensions like factuality or coherence without supervised training on human judgment data.

Comparison of text evaluation paradigms

The authors tackle this gap by asking: What if we evaluate generated text by directly measuring how easily a pre-trained model can generate it? As illustrated above, this shifts the paradigm from matching, regression, or ranking to a pure generation task.

Methods

The authors propose BARTScore, an unsupervised metric built on the pre-trained sequence-to-sequence model BART. Instead of extracting embeddings to compute cosine similarity, BARTScore calculates the weighted log probability of one text ($\mathbf{y}$) given another text ($\mathbf{x}$).

Because it is framed as a generation task, BARTScore can be flexibly adapted to evaluate different perspectives simply by changing the input and output:

  • Faithfulness ($s \to h$): Probability of generating the hypothesis from the source text (great for factuality and relevance).
  • Precision ($r \to h$): Probability of generating the hypothesis from the reference text.
  • Recall ($h \to r$): Probability of generating the reference from the hypothesis (great for semantic coverage).
  • F-score ($r \leftrightarrow h$): The arithmetic average of Precision and Recall.

The authors also introduce two powerful extensions:

  1. Prompting: Adding short phrases (e.g., prepending "in summary" or "such as") to the input or output to bring the evaluation task closer to the model's pre-training distribution.
  2. Fine-tuning: Updating the underlying BART model on downstream generation tasks (like summarization on the CNN/DM dataset or paraphrasing on ParaBank2) to align the model's domain with the evaluation task.

Datasets

The authors evaluated BARTScore across 16 datasets covering three main tasks and 7 different evaluation perspectives (Informativeness, Relevance, Fluency, Coherence, Factuality, Semantic Coverage, and Adequacy):

  • Machine Translation (MT): WMT19 metrics shared task (7 language pairs).
  • Text Summarization: REALSumm, SummEval, and NeR18.
  • Factuality: Rank19 and QAGS (QAGS-C, QAGS-X).
  • Data-to-Text (D2T): BAGEL, SFHOT, and SFRES.

Key Insights and Ideas

  • Evaluation as Generation is Highly Effective: By utilizing the entire encoder-decoder architecture, BARTScore better leverages the knowledge embedded during pre-training compared to metrics that only use encoder representations.
  • Prompts are "Zero-Shot" Performance Boosters: Simply adding a natural language prompt to the text can significantly improve a metric's correlation with human judgment without requiring any parameter updates or supervised training.
  • Domain Adaptation Works for Metrics Too: Fine-tuning the base model on a task related to the evaluation (e.g., training BART on a summarization dataset before using it to evaluate summaries) makes the metric a much harsher, more accurate judge of quality.
  • Robustness to SOTA Systems: As text generation models get better, older metrics struggle to distinguish between "good" and "great" outputs. BARTScore maintains a much more stable correlation with human judgments even when evaluating top-tier systems.

Key Results

  • Broad Superiority: BARTScore outperformed existing top-scoring metrics in 16 out of 22 test settings across the 16 datasets.
  • Massive Gains in Summarization: On the SummEval dataset, vanilla BARTScore outperformed BERTScore by 0.251 and MoverScore by 0.265 (Spearman correlation).
  • Near-Human Factuality Evaluation: On the Rank19 factuality dataset, BARTScore (fine-tuned on CNN/DM) achieved an accuracy of 0.836, crushing the previous SOTA (FactCC at 0.700) and nearly matching the human baseline of 0.839.
  • Prompting Magic: In German-to-English Machine Translation, simply adding the prompt "such as" improved Kendall’s Tau correlation by 0.033, allowing this unsupervised metric to surpass even heavily supervised SOTA metrics like BLEURT and COMET.

Limitations and Future Work

  • System Bias: The authors note that BARTScore is much better at distinguishing the quality of abstractive summarization systems than extractive ones. It tends to rank extractive systems slightly lower than human judges do.
  • Prompt Unpredictability: While prompting consistently improves metrics related to semantic overlap (informativeness, coverage), its effect on factuality and linguistic quality is less predictable and sometimes degrades performance.
  • Future Work: The authors suggest exploring automated prompt search strategies to find optimal phrases for evaluation. They also highlight a future where metrics and systems "co-evolve"—as sequence-to-sequence models improve, they can be directly plugged in to serve as increasingly reliable automated metrics.

Summary updated

Abstract

A wide variety of NLP applications, such as machine translation, summarization, and dialog, involve text generation. One major challenge for these applications is how to evaluate whether such generated texts are actually fluent, accurate, or effective. In this work, we conceptualize the evaluation of generated text as a text generation problem, modeled using pre-trained sequence-to-sequence models. The general idea is that models trained to convert the generated text to/from a reference output or the source text will achieve higher scores when the generated text is better. We operationalize this idea using BART, an encoder-decoder based pre-trained model, and propose a metric BARTScore with a number of variants that can be flexibly applied in an unsupervised fashion to evaluation of text from different perspectives (e.g. informativeness, fluency, or factuality). BARTScore is conceptually simple and empirically effective. It can outperform existing top-scoring metrics in 16 of 22 test settings, covering evaluation of 16 datasets (e.g., machine translation, text summarization) and 7 different perspectives (e.g., informativeness, factuality). Code to calculate BARTScore is available at https://github.com/neulab/BARTScore, and we have released an interactive leaderboard for meta-evaluation at http://explainaboard.nlpedia.ai/leaderboard/task-meval/ on the ExplainaBoard platform, which allows us to interactively understand the strengths, weaknesses, and complementarity of each metric.
BibTeX
@inproceedings{yuan2021bartscore,
  author = {Weizhe Yuan and Graham Neubig and Pengfei Liu},
  title = {{BARTScore: Evaluating Generated Text as Text Generation}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=5Ya8PbvpZ9},
}
Figure