Dialogue-Oriented Review Summary Generation for Spoken Dialogue Recommendation Systems

Jingjing Liu, Stephanie Seneff, Victor ZueNAACL 2010 · OpenReview

TL;DR

This paper tackles the challenge of generating concise, informative review summaries for spoken dialogue systems, moving beyond simple text summarization. They propose a three-level framework and a decision tree-based approach to automatically identify and select the most relevant phrases from reviews, achieving human-comparable accuracy and significantly outperforming existing methods.

What Problem Does This Paper Address?

Current spoken dialogue systems are becoming more sophisticated, aiming to provide subjective recommendations rather than just factual answers. To achieve this, they need to leverage the vast amount of user-generated reviews available online. However, directly applying text-based summarization techniques to dialogue systems is problematic. Dialogue turns are short and speech is serial, making lengthy summaries impractical. The authors identify a gap in creating fine-grained, informative, yet concise review summaries specifically tailored for human-computer conversations. They aim to bridge the gap between raw review data and helpful, conversational recommendations.

Methods

The paper proposes a three-level framework for review-based recommendation dialogue systems:

  1. Linguistic Phrase Extraction: (Previously explored by the authors) This involves parsing review sentences to extract phrases, including adjective-noun relationships, adverbs, and negations, and assigning sentiment scores. This approach is designed to capture rare but informative phrases.
  2. Dialogue-Oriented Review Summary Generation: This is the core contribution of this paper. They address the challenge of filtering out low-quality or irrelevant phrases from the extracted set to create a concise summary database.
  3. Phrase Classification: They frame the problem as classifying phrases as "highly informative and relevant" (y=1) or "uninformative" (y=-1).
  4. Feature Engineering: They use a rich set of features, including:
  5. Statistical Features: Unigram/bigram probabilities of words and adjective-noun pairs.
  6. Sentiment Features: Sentiment scores of phrases.
  7. Semantic Features: Derived from a context-related ontology, indicating if a phrase's topic belongs to a generic semantic category (e.g., "food," "service," "atmosphere").
  8. Classification Models: They train and compare Support Vector Machines (SVMs) and Decision Trees (DTs).
  9. Human-Friendly Dialogue Generation: (Briefly mentioned) This level involves using the generated summary database to create natural-sounding conversational responses.

Datasets

  • Restaurant Reviews: A dataset of 137,569 reviews for 24,043 restaurants in 9 U.S. cities was harvested from an online restaurant evaluation website.
  • Training Data: 3,000 phrases extracted from "pros/cons" sections of reviews.
  • Test Data: A separate set of 3,000 phrases extracted from review texts.
  • Restaurant Menus: A collection of 16,141 restaurant menus from an online resource was used to build a context-related ontology for semantic feature extraction.

Key Insights and Ideas

  • Dialogue vs. Text Summarization: The fundamental difference lies in the constraints of dialogue turns (short, serial) versus text (skippable, scannable). This necessitates a focus on short, succinct phrases.
  • Subjectivity of "Informativeness": What constitutes an "informative" phrase is subjective. The authors found that common phrases like "great value" are often uninformative, while specific, domain-relevant phrases (e.g., "absolutely delicious spaghetti carbonara") are highly valued, even if rare.
  • Decision Trees for Subjective Tasks: Decision trees are particularly well-suited for this task because they can effectively combine Boolean (e.g., ontology membership) and continuous (e.g., sentiment score) features, mirroring human "hierarchical if-else" decision-making.
  • Adjective-Noun Pair Importance: While individual adjectives might not be dominant, the bigram probability of an adjective-noun pair is a crucial feature for determining phrase relevance. This highlights the importance of context.
  • Semantic Features are Key: Incorporating semantic features derived from ontologies significantly improves classification performance, demonstrating the need to understand the meaning of phrases beyond their statistical co-occurrence.

Key Results

  • Decision Tree Outperforms SVM and Baseline: The decision tree model achieved the highest precision, outperforming SVMs by 13% and a heuristic rule baseline by 36% on average.
  • Human-Comparable Accuracy: The decision tree model achieved a precision of 77.9% and 74.5% on two annotation sets, which is comparable to human judgment (74% kappa agreement between annotators).
  • Feature Contributions:
    • Semantic features contributed significantly, reducing precision by 7.3-9.1% when removed.
  • The adjective-noun pair bigram probability was the most influential feature, with its removal leading to the lowest precision.
  • Sentiment features also played a significant role, with their removal dropping precision by 7.9-14.5%.

Limitations and Future Work

  • Limitations:
    • The paper focuses on the restaurant domain, and while the approach is designed to be domain-independent, its effectiveness in other domains needs further validation.
  • The heuristic rule baseline, while complex, was manually created, suggesting that hand-crafted rules can still be competitive if well-designed.
  • Future Work:
    • Extending sentiment scoring to noun/verb sentiment assessment.
  • Applying the review summary generation approach to other domains and languages.
  • Collecting data on user engagement with dialogue systems that use these summaries to evaluate their effectiveness.
  • Further research into the generation of human-friendly dialogue responses from the summary database.

Summary updated

Abstract

In this paper we present an opinion summarization technique in spoken dialogue systems. Opinion mining has been well studied for years, but very few have considered its application in spoken dialogue systems. Review summarization, when applied to real dialogue systems, is much more complicated than pure text-based summarization. We conduct a systematic study on dialogue-system-oriented review analysis and propose a three-level framework for a recommendation dialogue system. In previous work we have explored a linguistic parsing approach to phrase extraction from reviews. In this paper we will describe an approach using statistical models such as decision trees and SVMs to select the most representative phrases from the extracted phrase set. We will also explain how to generate informative yet concise review summaries for dialogue purposes. Experimental results in the restaurant domain show that the proposed approach using decision tree algorithms achieves an outperformance of 13% compared to SVM models and an improvement of 36% over a heuristic rule baseline. Experiments also show that the decision-tree-based phrase selection model can achieve rather reliable predictions on the phrase label, comparable to human judgment. The proposed statistical approach is based on domain-independent learning features and can be extended to other domains effectively.
BibTeX
@inproceedings{liu2010dialogue,
  author = {Jingjing Liu and Stephanie Seneff and Victor Zue},
  title = {{Dialogue-Oriented Review Summary Generation for Spoken Dialogue Recommendation Systems}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=By4nc7bd-S},
}
Figure