Semantic Similarity Based Evaluation for Abstractive News Summarization

Figen Beken Fikri, Kemal Oflazer, Berrin YanikogluACL 2021

TL;DR

This paper tackles the limitations of ROUGE, a common metric for evaluating news summarization, which struggles with abstractive summaries that generate novel sentences. The authors introduce a novel semantic similarity-based evaluation metric for Turkish abstractive summarization, leveraging a new Turkish semantic textual similarity dataset (STSb-TR) and advanced language models. Their approach demonstrates better alignment with human judgments than ROUGE, particularly for abstractive summaries.

What Problem Does This Paper Address?

Evaluating the quality of automatically generated text, especially summaries, is a significant challenge in Natural Language Processing (NLP). The dominant metric for summarization evaluation, ROUGE, relies on lexical overlap between the generated summary and human-written "gold standard" summaries. While effective for extractive summarization (which selects sentences directly from the source), ROUGE falters when evaluating abstractive summarization. Abstractive systems aim to generate novel sentences that capture the essence of the source text, often using different wording. This is especially problematic for agglutinative languages like Turkish, where word formation can lead to a vast vocabulary and a high type/token ratio, making direct word overlap a poor indicator of semantic equivalence. The paper addresses this gap by proposing a semantic similarity-based evaluation metric that can better assess the quality of abstractive summaries, particularly for Turkish.

Methods

The core of the paper's methodology involves developing and applying semantic similarity models for evaluating Turkish news summaries. They achieve this through two main avenues:

  • Creating a Turkish Semantic Textual Similarity (STS) Dataset: The authors translate the English Semantic Textual Similarity benchmark (STSb) dataset into Turkish, creating the STSb-TR dataset. This dataset is crucial for training and evaluating semantic similarity models for Turkish.
  • Leveraging and Adapting Language Models: They experiment with various state-of-the-art language models for sentence representation and cross-encoding. These include:
  • Sentence Representation Models (Bi-Encoders): LASER, LaBSE, MUSE (CNN and Transformer variants), BERT (specifically BERTurk and M-BERT), XLM-R, and Sentence-BERT. These models map sentences into fixed-size vectors, and similarity is computed using cosine similarity.
  • Cross-Encoders: BERTurk, M-BERT, and XLM-R are also used in a cross-encoder setup, where both sentences are fed into the network to directly predict a similarity score.
  • Training and Fine-tuning: Models are trained on both the NLI-TR (Natural Language Inference for Turkish) dataset and the newly created STSb-TR dataset. They observe that pre-training on NLI-TR and then fine-tuning on STSb-TR generally improves performance.

Datasets

  • STSb-TR: This is a newly introduced dataset created by translating the English Semantic Textual Similarity benchmark (STSb) dataset into Turkish. It serves as the primary benchmark for evaluating semantic textual similarity models in Turkish.
  • NLI-TR: This dataset, a translation of the English SNLI and MultiNLI datasets, is used for pre-training language models for natural language inference tasks in Turkish.
  • MLSUM: The authors use the MLSUM dataset for their summarization experiments. This is a large-scale multilingual summarization dataset that includes a Turkish subset, compiled from news articles and their summaries.

Key Insights and Ideas

  • ROUGE's Inadequacy for Abstractive Summaries: The paper strongly highlights how ROUGE's reliance on lexical overlap makes it a poor evaluator for abstractive summarization, especially for morphologically rich languages like Turkish. The example of "aramak istiyorum" vs. "telefon etmek istiyorum" vividly illustrates this.
  • Semantic Similarity as a Better Metric: The core idea is that semantic similarity, which focuses on meaning rather than exact word matches, is a more appropriate metric for evaluating abstractive summaries.
  • Cross-Encoders Outperform Bi-Encoders: The results indicate that cross-encoder models generally achieve higher correlations with human judgments compared to bi-encoder models, suggesting that jointly processing sentence pairs leads to better similarity assessment.
  • NLI Pre-training Boosts Performance: Pre-training semantic similarity models on NLI-TR data significantly improves their performance on the STSb-TR dataset, demonstrating the benefit of leveraging inference tasks for similarity learning.
  • New Turkish STS Dataset is High Quality: The authors assert that their translated STSb-TR dataset is of high quality and does not require costly human annotation, making it a valuable resource for the Turkish NLP community.

Key Results

  • The best performing semantic similarity models (specifically cross-encoder models trained on NLI-TR and then STSb-TR) achieved significantly better Pearson and Spearman correlations with human judgments (relevance, consistency, fluency, and average) compared to ROUGE and BERTScore.
  • For example, the BERTurk+NLI+STS cross-encoder model achieved a Pearson correlation of 59.98 and Spearman of 59.17 with human judgments, outperforming ROUGE-L (41.83 Pearson, 41.95 Spearman) and BERTScore (45.49 Pearson, 45.75 Spearman).
  • Qualitative analysis showed that their proposed metrics could capture semantic similarity in cases where ROUGE failed, particularly when summaries used different wording but conveyed the same meaning.
  • Interestingly, an extractive baseline (Lead-3) performed better in terms of semantic similarity scores than the abstractive mT5 model, though mT5 still outperformed extractive baselines on ROUGE and BERTScore.

Limitations and Future Work

  • Translation Errors: The authors acknowledge that translation errors might exist in the STSb-TR dataset, but they assume these do not significantly impact the overall results due to high BLEU scores achieved during translation quality assessment.
  • Model Performance Variation: While some models showed strong performance, others, particularly BERT models without specific STS training, performed poorly.
  • Future Work: The paper suggests that their semantic similarity models can be applied to evaluate abstractive summarization in Turkish. Further research could explore more sophisticated cross-lingual semantic similarity models and their application to other abstractive text generation tasks.

Summary updated

BibTeX
@inproceedings{fikri2021semantic,
  author = {Figen Beken Fikri and Kemal Oflazer and Berrin Yanikoglu},
  title = {{Semantic Similarity Based Evaluation for Abstractive News Summarization}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure