UniCOQE: Unified Comparative Opinion Quintuple Extraction As A Set

Zinong Yang, Feng Xu, Jianfei Yu, Rui XiaACL 2023 · DOI

Introduction

Comparative Opinion Quintuple Extraction (COQE) is a critical task in sentiment analysis that aims to identify and extract structured information from comparative statements in text. A complete extraction requires identifying five distinct elements, known as a quintuple: the Subject, the Object, the Aspect being compared, the Opinion expression, and the final Preference (e.g., BETTER or WORSE).

The COQE task involves extracting five distinct elements from comparative text to form structured quintuples.

Traditionally, this task has been approached using pipeline models that break the extraction into sequential sub-tasks. However, these methods suffer from error propagation, where a mistake in early entity recognition compromises the entire quintuple. While recent generative models like T5 offer an end-to-end alternative, they introduce a new challenge: "false order bias."

The Challenge: False Order Bias in Generative Models

Generative models typically treat extraction as a sequence-to-sequence task, requiring the model to output quintuples in a specific order. However, in sentences containing multiple comparisons, there is no inherent natural order. Forcing a model to learn an arbitrary sequence (such as alphabetical or appearance-based) creates a "false order bias." This confuses the model during training, as it is penalized for the order of its output rather than the accuracy of the extracted content.

The UniCOQE Framework

To address this, the authors propose UniCOQE, a framework that treats COQE as a set-prediction problem. Built upon the T5 architecture, UniCOQE utilizes a novel Predict-and-Assign training paradigm that decouples the content of the extraction from its sequence.

The Predict-and-Assign Paradigm

The core innovation lies in how the model is trained:

  1. Unified Template: The model is prompted to generate all quintuples in a single string, using semicolons as delimiters.
  2. Assigning Stage: During training, the model first generates a set of predicted quintuples. Instead of comparing these to a fixed-order ground truth, UniCOQE treats the ground truth as an unordered set.
  3. Hungarian Matching: The framework employs the Hungarian Algorithm to find the optimal one-to-one matching between predicted quintuples and "gold" (correct) quintuples. This matching is based on a cost matrix calculated using Intersection over Union (IoU) similarity.
  4. Dynamic Optimization: Once the best match is found, the ground truth is re-arranged to match the model's predicted order. The loss is then calculated based on this aligned sequence, ensuring the model is only penalized for extraction errors.
The Predict-and-Assign paradigm uses the Hungarian algorithm to align predictions with ground truth sets, eliminating order bias.

Experimental Results

UniCOQE was evaluated on three benchmark datasets: Camera-COQE (English), Car-COQE (Chinese), and Ele-COQE (Chinese). The model achieved a new state-of-the-art (SOTA) across all metrics, with particularly dramatic improvements in the "Exact Match" F1 score for quintuples:

  • Camera-COQE: Increased from a previous SOTA of 13.36 to 31.95.
  • Car-COQE: Improved from 29.75 to 36.55.
  • Ele-COQE: Improved from 30.73 to 35.46.

Ablation studies confirmed that the set-matching strategy was the primary driver of these gains, especially in complex sentences containing multiple comparisons where standard generative models typically struggle.

Conclusion

UniCOQE demonstrates that treating information extraction as a set-prediction task significantly improves performance by removing the artificial constraints of sequential output. By allowing the ground truth to dynamically align with the model's internal logic, UniCOQE provides a cleaner training signal and sets a new benchmark for comparative opinion mining. Future work will explore applying this "predict-and-assign" logic to other complex extraction tasks such as event extraction and cross-domain sentiment analysis.

Summary updated

BibTeX
@inproceedings{yang2023unicoqe,
  author = {Zinong Yang and Feng Xu and Jianfei Yu and Rui Xia},
  title = {{UniCOQE: Unified Comparative Opinion Quintuple Extraction As A Set}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
  doi = {10.18653/v1/2023.findings-acl.775},
}
Figure