Evaluation of Review Summaries via Question-Answering

Nannan Huang, Xiuzhen ZhangALTA 2021

Introduction

In the field of Natural Language Processing (NLP), opinion summarization—the task of condensing numerous consumer reviews into a concise summary—has seen significant progress. However, evaluating the quality of these summaries remains a major bottleneck. Traditional metrics like ROUGE rely on n-gram overlap, which often fails to capture semantic nuances or sentiment consistency. For instance, a summary that flips the sentiment of a review (e.g., "the battery is great" vs. "the battery is terrible") might still receive a high ROUGE score due to word overlap.

To address these limitations, this research introduces RunQA (Review Summary Evaluation via Question Answering), a novel metric designed to evaluate how well machine-generated summaries capture the specific opinions and intents found in human-written references.

The RunQA Framework

RunQA shifts the evaluation paradigm from word-matching to information verification through a question-answering (QA) lens. The framework operates in four distinct stages, as illustrated in the pipeline below:

Figure 1: The RunQA evaluation pipeline, showing the transition from reference summaries to a final RunQA score via question generation and answering.
  1. Answer Selection: Unlike previous QA metrics that focus primarily on Noun Phrases (NPs), RunQA identifies "opinion-bearing" spans, including adjectives, adverbs, and verbs.
  2. Question Generation: A BART model, fine-tuned on the QA2D dataset, converts these selected spans into natural language questions.
  3. Answering: A pre-trained ELECTRA-Large model (fine-tuned on SQuAD 2.0) attempts to answer these questions using the system-generated summary as the context.
  4. Verification (LERC): To account for paraphrasing, the system employs LERC (Learned Evaluation metric for Reading Comprehension) to compare the generated answers against the original "gold" answers, providing a more flexible and accurate similarity score than standard F1 metrics.

The Importance of Opinion-Bearing Spans

A core insight of this work is that in the domain of product reviews, nouns alone are insufficient for capturing the "meat" of the information. Analysis of Summarization Content Units (SCUs) reveals that adjectives and verbs are critical components of consumer opinions.

Figure 2: Distribution of Part-of-Speech (POS) tags in Summarization Content Units, highlighting the high frequency of adjectives and verbs compared to traditional noun-centric approaches.

As shown in the figure above, while nouns are frequent, adjectives and verbs constitute a massive portion of the content that humans deem important in a summary. By targeting these specific parts of speech, RunQA ensures that the evaluation focuses on the evaluative language that defines a review.

Experimental Results

The researchers benchmarked RunQA using the Amazon product review dataset, comparing it against human judgments of Coverage (recall) and Focus (precision). RunQA significantly outperformed existing standards:

  • Human Correlation: RunQA achieved a Pearson correlation of 0.597 for Coverage and 0.714 for Focus, comfortably beating ROUGE-1 (0.479) and BERTScore (0.651) respectively.
  • Discriminative Power: In a ranking task designed to distinguish human-written summaries from machine-generated ones, RunQA (LERC) achieved 91.67% accuracy. In contrast, BERTScore performed at 54.44%, indicating it struggled to differentiate between high-quality human summaries and machine outputs.

Conclusion

RunQA represents a significant step forward in the evaluation of opinionated text. By leveraging a QA-based approach that prioritizes adjectives and verbs, it provides a more robust and human-aligned metric than traditional n-gram or general semantic similarity models. Future work will explore the metric's applicability across broader domains, such as movie and restaurant reviews, and investigate ways to reduce the computational overhead associated with neural QA pipelines.

Summary updated

BibTeX
@inproceedings{huang2021evaluation,
  author = {Nannan Huang and Xiuzhen Zhang},
  title = {{Evaluation of Review Summaries via Question-Answering}},
  booktitle = {ALTA},
  year = {2021},
}
Figure