Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics

Gisela Vallejo, Timothy Baldwin, Lea FrermannALTA 2022

Abstract

Traditional text similarity metrics often fail to capture semantic meaning in long documents. While question-answering (QA) based metrics like QuestEval offer a promising alternative, their validation typically relies on Pearson correlation—a metric highly susceptible to outlier inflation. This paper investigates QuestEval's performance across various datasets, demonstrating that high correlation scores can mask poor performance on the majority of data points. We advocate for the use of distribution-aware validation techniques, such as Kernel Density Estimation (KDE) plots, to ensure metric reliability.

Introduction

Evaluating document similarity is a cornerstone of NLP tasks like fact-checking and summarization. Traditional overlap-based metrics (e.g., ROUGE, BLEU) are increasingly viewed as insufficient for capturing factual consistency. Consequently, QA-based metrics have emerged, operating on the principle that if Document B can answer questions derived from Document A, they are semantically similar. However, the standard validation protocol—correlating these metrics against human judgments—may be fundamentally flawed.

Methodology

The authors evaluate QuestEval, a pipeline that generates questions from source documents and attempts to answer them using target documents. To "evaluate the examiner," QuestEval scores were compared against human-annotated gold standards across three datasets: SummEval (long-to-short), ABC News (long-to-long), and SemEval 2022 Task 8 (complex long-to-long). Crucially, the study moves beyond raw correlation, employing Median Absolute Deviation (MAD) for outlier detection and Kernel Density Estimation (KDE) plots to visualize score distributions.

The Outlier Inflation Trap

A primary finding is the extreme sensitivity of Pearson correlation ($r$) to outliers. In the ABC News dataset, an initial correlation of 0.33 was observed. However, KDE visualization (Figure 1) shows that the bulk of the data is clustered at low similarity scores, with the regression line heavily influenced by a small number of extreme cases.

KDE plot for ABC News showing how a dense cluster at low scores can lead to misleading correlation results.

When only 20 outliers (out of 1,225 pairs) were removed, the correlation plummeted to 0.11—a 66% drop. This indicates that the metric failed to distinguish similarity for the vast majority of the dataset, despite a seemingly "acceptable" initial $r$ score.

Robustness in Long-Form Similarity

Despite validation concerns, QuestEval demonstrates significant potential for long-document analysis. On the SemEval 2022 dataset, the metric achieved a robust correlation of $r=0.77$. Unlike the ABC News case, this correlation remained stable even after outlier removal, suggesting a genuine capture of document similarity across the entire distribution (Figure 2).

KDE plot for SemEval 2022 Task 8, demonstrating a robust and linear relationship between QuestEval and human annotations.

Conclusion

The study concludes that a single correlation coefficient is insufficient for validating NLP metrics. Researchers must account for data distribution and skewness. While QuestEval proves versatile and robust for certain long-form tasks—particularly in measuring factual consistency—its "state-of-the-art" status in other contexts may be a statistical mirage driven by outliers. Future work should focus on multilingual expansion and more nuanced question-generation strategies to improve sensitivity to the "bulk" of textual data.

Summary updated

BibTeX
@inproceedings{vallejo2022evaluating,
  author = {Gisela Vallejo and Timothy Baldwin and Lea Frermann},
  title = {{Evaluating the Examiner: The Perils of Pearson Correlation for Validating Text Similarity Metrics}},
  booktitle = {ALTA},
  year = {2022},
}
Figure