PASS: Perturb-and-Select Summarizer for Product Reviews

Nadav Oved, Ran LevyACL 2021

TL;DR

This paper introduces PASS, a novel approach to product review summarization that tackles the twin challenges of generic, uninformative summaries and self-contradictory outputs. By perturbing input reviews and then selecting the best summary based on coherence, PASS generates more informative and diverse summaries that better reflect the nuances of customer opinions.

What Problem Does This Paper Address?

The task of automatically summarizing product reviews is crucial for helping consumers make informed decisions. However, existing summarization systems often fall short in two key areas:

  1. Lack of Informativeness and Diversity: Many systems tend to produce generic, template-like summaries that could apply to almost any product. This is partly because they favor content that appears frequently across many reviews, leading to a lack of specific detail. Furthermore, summaries generated for different products often exhibit high similarity, failing to capture the unique opinions associated with each item. The authors introduce "CP-Diversity" (Cross-Product Diversity) to quantify this issue.
  2. Self-Contradiction and Inconsistency: Product reviews themselves can be contradictory. For instance, one reviewer might praise a product's ease of use, while another might find it difficult. Standard summarization models, when faced with such conflicting information, can produce summaries that contradict themselves, making them unreliable. The authors highlight the need for "self-consistency" in summaries, meaning the summary itself should not contain conflicting statements.

Methods

The proposed system, PASS (Perturb-and-Select Summarizer), is built upon a large pre-trained Transformer model (T5) and employs a few-shot fine-tuning approach. Its core innovation lies in two stages:

  1. Perturb-and-Select Summarization:
    • Input Perturbation (LkO): During inference, PASS systematically perturbs the input by "leaving out" a subset of k reviews (LkO stands for "leave-k-out"). This process, applied to various combinations of reviews, generates multiple candidate summaries for a single product. This is crucial because it allows the model to explore different facets of opinion that might be overshadowed if all reviews were always considered equally. The authors found that applying this perturbation during inference, rather than training, leads to higher output variation.
  2. Candidate Summary Generation: The perturbed inputs are fed into a fine-tuned T5 model, which generates a diverse set of candidate summaries.
  3. Candidate Summary Ranking:
    • Coherence-Based Ranking: The generated candidate summaries are then ranked to select the best one. The primary ranking criterion is coherence, aiming to filter out self-contradictory or poorly formed summaries.
  4. Pairwise Summary Classifier: To achieve this ranking, the authors train a pairwise classifier (also based on T5) that takes two summaries and determines which one is more coherent. This classifier is trained on human-annotated coherence scores.
  5. Tie-Breaking: A secondary classifier, trained to assess fluency, is used to break ties when summaries are equally coherent.

Datasets

  • Training and Evaluation: The primary dataset used is a publicly available Amazon product reviews summarization dataset (from Brazinskas et al., 2020a), containing reviews and reference summaries for 60 products. A similar dataset from Yelp businesses is also used for end-to-end evaluation.
  • Coherence Classifier Training: A public dataset of human-annotated summaries from 16 different summarization models applied to 100 news articles (CNN/DailyMail dataset) is used to train the pairwise coherence and fluency classifiers. The authors specifically use expert annotations from this dataset.

Key Insights and Ideas

  • Input Perturbation as a Diversity Engine: The core idea of perturbing the input by selectively omitting reviews is a clever way to leverage the inherent diversity within product reviews. It allows the model to focus on specific subsets of opinions, leading to more varied and potentially more informative summaries.
  • Self-Consistency as a Primary Goal: The paper rightly identifies that for product reviews, self-consistency is more critical than factual consistency in the traditional sense. A summary that accurately reflects a range of opinions without contradicting itself is more valuable than one that might be factually correct but internally inconsistent.
  • Coherence as the Key Ranking Metric: By prioritizing coherence in the ranking stage, PASS effectively filters out problematic summaries that might arise from conflicting source material. This is a pragmatic approach to handling the inherent messiness of user-generated content.
  • Human-like Diversity is Achievable: The results suggest that PASS can generate summaries that are closer to human-generated summaries in terms of CP-Diversity than existing SOTA systems.

Key Results

  • Improved CP-Diversity and Coherence: PASS significantly outperforms baselines (CopyCat, FewSum, T5-FT) in coherence and CP-Diversity on both Amazon and Yelp datasets.
  • Comparable Informativeness: PASS maintains comparable informativeness (measured by ROUGE scores) to the best baselines, particularly T5-FT.
  • Reduced Crude Errors: PASS demonstrates a substantial reduction in crude coherence (CE) and self-consistency (SCE) errors compared to FewSum and T5-FT. For example, PASS has a CE-I error rate of 0.19 and an SCE-I error rate of 0.05, significantly lower than FewSum (0.50 and 0.30 respectively).
  • Effective Ranking: The pairwise coherence classifier achieves a reasonable accuracy of 0.61, with a low probability of critical errors. The fluency classifier further refines the ranking.

Limitations and Future Work

  • Domain Shift for Classifiers: The pairwise classifiers were trained on news articles, a different domain than product reviews. While the authors hypothesize that coherence and fluency are domain-agnostic, they acknowledge this as an area for further investigation.
  • Further Exploration of Perturbations: The authors suggest exploring other input perturbation methods and extending the framework to larger-scale inputs.
  • Promoting Other Linguistic Characteristics: Future work could focus on extending the PASS framework to promote other desired linguistic characteristics beyond coherence, such as novelty or specific sentiment nuances.
  • SPR Metric Validation: Further investigation into the proposed SPR metric for lexical diversity and its correlation with human judgments is planned.

Summary updated

BibTeX
@inproceedings{oved2021pass,
  author = {Nadav Oved and Ran Levy},
  title = {{PASS: Perturb-and-Select Summarizer for Product Reviews}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}

Loading related content...

Figure