Analyzing Coreference and Bridging in Product Reviews

Hideo Kobayashi, Christopher MalonCOLING 2022

TL;DR

This paper introduces a new dataset and annotation scheme for coreference and bridging in product reviews, addressing a critical gap in current Aspect-Based Sentiment Analysis (ABSA) and opinion summarization systems. They demonstrate that existing coreference systems and factuality checkers struggle with the unique challenges of product review discourse, highlighting the need for domain-specific solutions.

What Problem Does This Paper Address?

Modern e-commerce relies heavily on product reviews to inform consumer decisions and business strategies. Techniques like Aspect-Based Sentiment Analysis (ABSA) and opinion summarization aim to distill these reviews into actionable insights. However, these systems often overlook the complex linguistic phenomena of coreference (when different expressions refer to the same entity) and bridging (when an expression refers to something related to a previously mentioned entity). This oversight can lead to significant errors, such as misattributing sentiment about a competing product to the main product being reviewed, or generating factually incorrect summaries. Existing coreference systems, typically trained on general news text, don't perform well on the specific mention types found in product reviews, and current factuality scores fail to detect these subtle errors.

Methods

The authors propose a simplified mention classification task for product reviews. Instead of the complex task of identifying all coreference and bridging relations between mention pairs, they focus on classifying individual mentions into nine types: "Main Product," "Competing Product," "Interacting Item," and their respective "Part-of/Attribute-of" variants, along with a "Generic Term" and "Others." This approach aims to reduce the annotation burden while still capturing the essential information needed to resolve coreference and bridging. They collected annotations from crowdworkers, ensuring good inter-annotator agreement.

Datasets

The authors introduce a new annotated dataset of 498 electronics reviews from the Amazon Review Dataset (McAuley et al., 2015; He and McAuley, 2016). This dataset contains 8,894 mentions annotated with the proposed mention types. They also leverage existing datasets for training their sentiment analysis classifier, including electronics reviews from the Amazon review dataset and a sentence-level sentiment dataset from SemEval 2016 Task 5.

Key Insights and Ideas

  • Product reviews have unique discourse challenges: The complexity of coreference and bridging in product reviews, involving main products, competitors, and interacting items, is not adequately handled by general-purpose NLP systems.
  • Existing factuality scores are insufficient: State-of-the-art Natural Language Inference (NLI)-based factuality scores, like SummaC-ZS, often fail to detect coreference errors in product reviews, even at the document level.
  • General coreference models struggle with product mentions: Coreference systems trained on general domains (like OntoNotes) show significantly lower performance when applied to product review mentions, suggesting a need for domain adaptation or specialized models.
  • Simplified annotation is effective: A mention classification task, rather than full relation labeling, can yield high-quality annotations with good inter-annotator agreement from crowdworkers.

Key Results

  • The authors collected annotations for 8,894 mentions across 498 product reviews.
  • Inter-annotator agreement, measured by Cohen's kappa, was substantial at .681 for agreed mentions.
  • When testing an OntoNotes-based coreference system (Xu and Choi, 2020), they found significantly lower average F1 scores for "Competing" and "Generic" mention types compared to "Main" product mentions. The overall AVG F1 was lower than that achieved on OntoNotes.
  • The SummaC-ZS factuality score, even at document granularity, showed limited success in detecting inconsistent entity substitutions in product reviews, achieving an AUC of .770.

Limitations and Future Work

The authors acknowledge that the current dataset size, while useful for evaluation, may not be sufficient for training complex ABSA or NLI systems. They plan to continue collecting data to make it more suitable for development. They envision future work where ABSA and NLI systems can leverage these predicted mention types as features to improve accuracy and avoid entity reference errors. They also note that some disagreements in their factuality testing arose from the inherent ambiguity of human interpretation or annotation errors, suggesting that even with perfect coreference resolution, some ambiguity might persist.

Summary updated

BibTeX
@inproceedings{kobayashi2022analyzing,
  author = {Hideo Kobayashi and Christopher Malon},
  title = {{Analyzing Coreference and Bridging in Product Reviews}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
}
Figure