Improving Specificity in Review Response Generation with Data-Driven Data Filtering

Tannon Kew, Martin VolkACL 2022

TL;DR

This paper tackles the problem of generic responses generated by neural models for online reviews. By developing and applying data-driven filtering techniques to training data, the authors significantly improve the specificity of generated responses, making them more useful and relevant to the original reviews.

What Problem Does This Paper Address?

Automated generation of responses to online customer reviews is a valuable tool for businesses. While neural text generation models can produce fluent and natural-sounding responses, they often suffer from a "one-size-fits-all" problem, generating overly generic replies that lack specific details from the review. This genericness can limit the utility of these systems, especially in contexts where personalized and specific feedback is crucial. The paper aims to address this by identifying and filtering out generic responses from the training data to train models that produce more specific and informative replies.

Methods

The core of this work lies in developing methods to quantify the "genericness" of review responses. The authors propose three unsupervised scoring techniques:

  • Lexical Frequency: This method identifies words that are too common in the corpus, assuming they contribute little specific information. It calculates the ratio of high-frequency words to less frequent ones within a response.
  • Sentence Average: This approach considers larger semantic units by scoring sentences for genericness. It uses sentence embeddings to measure similarity against a pool of generic example sentences, then averages these scores across the response.
  • LM Perplexity: This method leverages a causal language model (distilled GPT-2) to score responses. Generic responses, being more predictable, are expected to have lower perplexity. The model is forced to score responses in isolation, maximizing surprisal for less generic texts.

These scoring methods are used to infer thresholds, and then a portion of the training data (specifically, the "best" 40%) is filtered based on these scores. The filtered datasets are then used to fine-tune BART-base models.

Datasets

  • Hospitality Review Data: A primary dataset of 500k unique hotel review-response pairs collected from TripAdvisor across seven countries. Approximately 90% was used for training, with 5% for validation and 5% for testing.
  • Mobile App Review Data: A dataset of mobile app review responses, originally introduced by Gao et al. (2019a), used for a generalizability study.

Key Insights and Ideas

  • Generic responses are a significant problem: The paper highlights that a large portion of available training data consists of generic responses, which negatively impact model performance.
  • Data filtering is effective: Removing generic responses from the training data, even with a substantial reduction in data size (e.g., 60% reduction), leads to models that generate more specific outputs.
  • Different scoring methods capture different aspects of genericness: Lexical frequency, sentence-level similarity, and language model perplexity each offer unique perspectives on what constitutes a generic response.
  • LM Perplexity shows promise: The LM PPL filtering method appears to strike a good balance between specificity and other desirable qualities.
  • Filtering can have side effects: While improving specificity, filtering might increase the occurrence of unsupported named entities or potentially misleading claims in generated responses.

Key Results

  • Automatic Metrics: Models trained on filtered data showed improvements across various metrics, particularly in chrF (computed against source reviews), indicating better specificity. Self-BLEU scores also improved, suggesting increased diversity.
  • Human Evaluation: Human evaluators, when asked to compare responses, consistently preferred those generated by models trained on filtered data, indicating improved specificity.
  • Data Reduction: The authors achieved significant improvements even when using only 40% of the original training data.
  • Generalizability: The filtering approach was shown to be effective on a different domain (mobile app reviews) as well.

Limitations and Future Work

  • Hallucinations: The paper acknowledges that filtering for specificity might lead to an increase in hallucinated content. Further work is needed to mitigate this and ensure factual accuracy.
  • Controlling Genericness: The current methods identify and filter generic responses, but they don't offer fine-grained control over the degree of genericness at inference time. Future work could explore using these scoring techniques to steer generation more directly.
  • Societal Impacts: The authors note the unknown societal impacts of deploying fully automatic response generation systems and emphasize the need for careful consideration of customer satisfaction and business relations.

Summary updated

BibTeX
@inproceedings{kew2022improving,
  author = {Tannon Kew and Martin Volk},
  title = {{Improving Specificity in Review Response Generation with Data-Driven Data Filtering}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
}
Figure