TL;DR
BARTScore flips the script on automated text evaluation by treating it as a text generation problem rather than a matching or regression task. By using pre-trained sequence-to-sequence models (like BART) to calculate the probability of generating a hypothesis from a source or reference text, BARTScore provides a highly flexible, unsupervised metric that achieves state-of-the-art correlation with human judgments across multiple NLP tasks and evaluation perspectives.
What Problem Does This Paper Address?
Evaluating the quality of generated text (like machine translation or summarization) is notoriously difficult. While human evaluation is the gold standard, it is expensive and slow. To automate this, the NLP community has recently shifted toward using pre-trained language models as evaluators (e.g., BERTScore, MoverScore).
However, there is a fundamental disconnect in how these metrics work: they use models pre-trained on text generation objectives merely as downstream feature extractors, completely ignoring the decoder and the generation head. Furthermore, most existing metrics are narrow, focusing only on one or two perspectives (like semantic equivalence or fluency) and struggling to evaluate complex dimensions like factuality or coherence without supervised training on human judgment data.
The authors tackle this gap by asking: What if we evaluate generated text by directly measuring how easily a pre-trained model can generate it? As illustrated above, this shifts the paradigm from matching, regression, or ranking to a pure generation task.
Methods
The authors propose BARTScore, an unsupervised metric built on the pre-trained sequence-to-sequence model BART. Instead of extracting embeddings to compute cosine similarity, BARTScore calculates the weighted log probability of one text ($\mathbf{y}$) given another text ($\mathbf{x}$).
Because it is framed as a generation task, BARTScore can be flexibly adapted to evaluate different perspectives simply by changing the input and output:
- Faithfulness ($s \to h$): Probability of generating the hypothesis from the source text (great for factuality and relevance).
- Precision ($r \to h$): Probability of generating the hypothesis from the reference text.
- Recall ($h \to r$): Probability of generating the reference from the hypothesis (great for semantic coverage).
- F-score ($r \leftrightarrow h$): The arithmetic average of Precision and Recall.
The authors also introduce two powerful extensions:
- Prompting: Adding short phrases (e.g., prepending "in summary" or "such as") to the input or output to bring the evaluation task closer to the model's pre-training distribution.
- Fine-tuning: Updating the underlying BART model on downstream generation tasks (like summarization on the CNN/DM dataset or paraphrasing on ParaBank2) to align the model's domain with the evaluation task.
Datasets
The authors evaluated BARTScore across 16 datasets covering three main tasks and 7 different evaluation perspectives (Informativeness, Relevance, Fluency, Coherence, Factuality, Semantic Coverage, and Adequacy):
- Machine Translation (MT): WMT19 metrics shared task (7 language pairs).
- Text Summarization: REALSumm, SummEval, and NeR18.
- Factuality: Rank19 and QAGS (QAGS-C, QAGS-X).
- Data-to-Text (D2T): BAGEL, SFHOT, and SFRES.
Key Insights and Ideas
- Evaluation as Generation is Highly Effective: By utilizing the entire encoder-decoder architecture, BARTScore better leverages the knowledge embedded during pre-training compared to metrics that only use encoder representations.
- Prompts are "Zero-Shot" Performance Boosters: Simply adding a natural language prompt to the text can significantly improve a metric's correlation with human judgment without requiring any parameter updates or supervised training.
- Domain Adaptation Works for Metrics Too: Fine-tuning the base model on a task related to the evaluation (e.g., training BART on a summarization dataset before using it to evaluate summaries) makes the metric a much harsher, more accurate judge of quality.
- Robustness to SOTA Systems: As text generation models get better, older metrics struggle to distinguish between "good" and "great" outputs. BARTScore maintains a much more stable correlation with human judgments even when evaluating top-tier systems.
Key Results
- Broad Superiority: BARTScore outperformed existing top-scoring metrics in 16 out of 22 test settings across the 16 datasets.
- Massive Gains in Summarization: On the SummEval dataset, vanilla BARTScore outperformed BERTScore by 0.251 and MoverScore by 0.265 (Spearman correlation).
- Near-Human Factuality Evaluation: On the Rank19 factuality dataset, BARTScore (fine-tuned on CNN/DM) achieved an accuracy of 0.836, crushing the previous SOTA (FactCC at 0.700) and nearly matching the human baseline of 0.839.
- Prompting Magic: In German-to-English Machine Translation, simply adding the prompt
"such as"improved Kendall’s Tau correlation by 0.033, allowing this unsupervised metric to surpass even heavily supervised SOTA metrics like BLEURT and COMET.
Limitations and Future Work
- System Bias: The authors note that BARTScore is much better at distinguishing the quality of abstractive summarization systems than extractive ones. It tends to rank extractive systems slightly lower than human judges do.
- Prompt Unpredictability: While prompting consistently improves metrics related to semantic overlap (informativeness, coverage), its effect on factuality and linguistic quality is less predictable and sometimes degrades performance.
- Future Work: The authors suggest exploring automated prompt search strategies to find optimal phrases for evaluation. They also highlight a future where metrics and systems "co-evolve"—as sequence-to-sequence models improve, they can be directly plugged in to serve as increasingly reliable automated metrics.
Summary updated