TL;DR
This paper enhances the unsupervised multi-review summarization model MeanSum by introducing "masks" derived from multi-aspect review segmentation to improve focus on key information. They also propose a length regularizer to combat repetitive summaries, achieving competitive results against baselines.
What Problem Does This Paper Address?
Generating concise summaries from multiple text documents (multi-document summarization) is a challenging task. While supervised methods exist, they require large, expensive datasets of paired documents and summaries. Unsupervised methods, like MeanSum, offer a promising alternative but have limitations. Specifically, MeanSum, while abstractive and competitive, struggles to handle multiple aspects within reviews and can produce repetitive summaries. This paper aims to address these shortcomings by making MeanSum more aspect-aware and better at controlling summary length.
Methods
The authors build upon the existing MeanSum model, which uses an auto-encoder for review representation and a summarization module trained with reconstruction and similarity losses. Their key innovations are:
- Multi-Aspect Masker (MAM): They introduce a method to segment reviews into five predefined aspects (service, cleanliness, value, location, room). These segmented parts, called "masks," are used as input to MeanSum. The idea is that these masks highlight more informative parts of the reviews, guiding the summarizer.
- Masking Strategies: They explore three ways to use these masks:
- Mask Concatenate: Simply concatenating all masks.
- RSAR (Regrouped Single Aspect Review): Grouping all segments belonging to the same aspect across all reviews for a hotel and training a single summarizer for each aspect.
- RSAR I-V: Similar to RSAR, but training an individual summarizer for each of the five aspects.
- Length Regularizer: To combat repetitive summaries and control output length, they introduce a "length loss" term into MeanSum's objective function. This regularizer penalizes summaries that are too short or too long, aiming for a more balanced output.
Datasets
- Hotel Review Dataset: A large dataset of approximately 1.3 million hotel reviews from TripAdvisor, used for training and evaluating the unsupervised models.
- TrustYou.com Dataset: A smaller dataset of around 83,850 hotel summaries (including overall and aspect-specific summaries) scraped from TrustYou.com, used to create a supervised baseline for comparison.
Key Insights and Ideas
- Masks Improve Focus: Using aspect-specific masks as input significantly improves MeanSum's performance, suggesting that focusing on salient aspects helps the model generate more relevant summaries.
- Aspect Distribution Matters: The uneven distribution of reviews across aspects (e.g., "Location" having many more reviews than others) is an intrinsic property of hotel reviews and influences summary characteristics.
- Length Control is Crucial: The proposed length regularizer effectively controls summary length, which is vital for avoiding repetition and potentially improving recall.
- Unsupervised vs. Supervised Trade-offs: The paper demonstrates that their enhanced unsupervised approach can achieve comparable or even better results than a supervised baseline, highlighting the value of unsupervised methods when labeled data is scarce.
Key Results
- The RSAR input strategy (grouping segments by aspect) yielded the best overall results, outperforming the original MeanSum on ROUGE-1 and Sentiment Accuracy, and achieving competitive ROUGE-2 and ROUGE-L scores.
- The length regularizer successfully reduced the average summary length, with parameters like
αandβallowing for fine-grained control. While it didn't directly improve ROUGE*F scores, it showed promise in balancing precision and recall by generating new content rather than just repeating. - The enhanced MeanSum model showed comparable performance to a supervised baseline on review and mask inputs, and was expected to surpass it when trained on RSAR.
Limitations and Future Work
- Language Fluency: A significant limitation identified is that concatenating masked segments can lead to fragmented language and reduced fluency, especially with the more granular RSAR inputs. This poses challenges for the language model.
- Parameter Tuning: The effectiveness of the length regularizer is highly dependent on parameter tuning. Further fine-tuning is needed to optimize its impact.
- Qualitative Length Control: The current length control is qualitative, meaning it can influence length but cannot strictly constrain it to a specific interval.
- Supervised Baseline Incomplete: The comparison with the supervised baseline was still in progress at the time of writing, particularly for the RSAR input.
- Future Work: The authors suggest further fine-tuning of parameters, exploring more sophisticated methods for length control, and investigating ways to improve language fluency when using masked inputs.
Summary updated