An Enhanced MeanSum Method For Generating Hotel Multi-Review Summarizations

Saibo Geng, Diego AntogniniarXiv 2020 · OpenReview

TL;DR

This paper enhances the unsupervised multi-review summarization model MeanSum by introducing "masks" derived from multi-aspect review segmentation to improve focus on key information. They also propose a length regularizer to combat repetitive summaries, achieving competitive results against baselines.

What Problem Does This Paper Address?

Generating concise summaries from multiple text documents (multi-document summarization) is a challenging task. While supervised methods exist, they require large, expensive datasets of paired documents and summaries. Unsupervised methods, like MeanSum, offer a promising alternative but have limitations. Specifically, MeanSum, while abstractive and competitive, struggles to handle multiple aspects within reviews and can produce repetitive summaries. This paper aims to address these shortcomings by making MeanSum more aspect-aware and better at controlling summary length.

Methods

The authors build upon the existing MeanSum model, which uses an auto-encoder for review representation and a summarization module trained with reconstruction and similarity losses. Their key innovations are:

  • Multi-Aspect Masker (MAM): They introduce a method to segment reviews into five predefined aspects (service, cleanliness, value, location, room). These segmented parts, called "masks," are used as input to MeanSum. The idea is that these masks highlight more informative parts of the reviews, guiding the summarizer.
  • Masking Strategies: They explore three ways to use these masks:
  • Mask Concatenate: Simply concatenating all masks.
  • RSAR (Regrouped Single Aspect Review): Grouping all segments belonging to the same aspect across all reviews for a hotel and training a single summarizer for each aspect.
  • RSAR I-V: Similar to RSAR, but training an individual summarizer for each of the five aspects.
  • Length Regularizer: To combat repetitive summaries and control output length, they introduce a "length loss" term into MeanSum's objective function. This regularizer penalizes summaries that are too short or too long, aiming for a more balanced output.

Datasets

  • Hotel Review Dataset: A large dataset of approximately 1.3 million hotel reviews from TripAdvisor, used for training and evaluating the unsupervised models.
  • TrustYou.com Dataset: A smaller dataset of around 83,850 hotel summaries (including overall and aspect-specific summaries) scraped from TrustYou.com, used to create a supervised baseline for comparison.

Key Insights and Ideas

  • Masks Improve Focus: Using aspect-specific masks as input significantly improves MeanSum's performance, suggesting that focusing on salient aspects helps the model generate more relevant summaries.
  • Aspect Distribution Matters: The uneven distribution of reviews across aspects (e.g., "Location" having many more reviews than others) is an intrinsic property of hotel reviews and influences summary characteristics.
  • Length Control is Crucial: The proposed length regularizer effectively controls summary length, which is vital for avoiding repetition and potentially improving recall.
  • Unsupervised vs. Supervised Trade-offs: The paper demonstrates that their enhanced unsupervised approach can achieve comparable or even better results than a supervised baseline, highlighting the value of unsupervised methods when labeled data is scarce.

Key Results

  • The RSAR input strategy (grouping segments by aspect) yielded the best overall results, outperforming the original MeanSum on ROUGE-1 and Sentiment Accuracy, and achieving competitive ROUGE-2 and ROUGE-L scores.
  • The length regularizer successfully reduced the average summary length, with parameters like α and β allowing for fine-grained control. While it didn't directly improve ROUGE*F scores, it showed promise in balancing precision and recall by generating new content rather than just repeating.
  • The enhanced MeanSum model showed comparable performance to a supervised baseline on review and mask inputs, and was expected to surpass it when trained on RSAR.

Limitations and Future Work

  • Language Fluency: A significant limitation identified is that concatenating masked segments can lead to fragmented language and reduced fluency, especially with the more granular RSAR inputs. This poses challenges for the language model.
  • Parameter Tuning: The effectiveness of the length regularizer is highly dependent on parameter tuning. Further fine-tuning is needed to optimize its impact.
  • Qualitative Length Control: The current length control is qualitative, meaning it can influence length but cannot strictly constrain it to a specific interval.
  • Supervised Baseline Incomplete: The comparison with the supervised baseline was still in progress at the time of writing, particularly for the RSAR input.
  • Future Work: The authors suggest further fine-tuning of parameters, exploring more sophisticated methods for length control, and investigating ways to improve language fluency when using masked inputs.

Summary updated

Abstract

Multi-document summaritazion is the process of taking multiple texts as input and producing a short summary text based on the content of input texts. Up until recently, multi-document summarizers are mostly supervised extractive. However, supervised methods require datasets of large, paired document-summary examples which are rare and expensive to produce. In 2018, an unsupervised multi-document abstractive summarization method(Meansum) was proposed by Chu and Liu, and demonstrated competitive performances comparing to extractive methods. Despite good evaluation results on automatic metrics, Meansum has multiple limitations, notably the inability of dealing with multiple aspects. The aim of this work was to use Multi-Aspect Masker(MAM) as content selector to address the issue with multi-aspect. Moreover, we propose a regularizer to control the length of the generated summaries. Through a series of experiments on the hotel dataset from Trip Advisor, we validate our assumption and show that our improved model achieves higher ROUGE, Sentiment Accuracy than the original Meansum method and also beats/ comprarable/close to the supervised baseline.
BibTeX
@misc{geng2020enhanced,
  author = {Saibo Geng and Diego Antognini},
  title = {{An Enhanced MeanSum Method For Generating Hotel Multi-Review Summarizations}},
  year = {2020},
  eprint = {2012.03656},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.03656},
}
Figure