Task-aware Contrastive Mixture of Experts for Quadruple Extraction in Conversations with Code-like Replies and Non-opinion Detection

Chenyuan He, Yuxiang Jia, Fei Gao, Senbin Zhu, Hongde Liu, Hongying Zan, Min PengEMNLP 2025

TL;DR

This paper introduces TaCoMoE, a novel LLM-based framework for dialogue aspect-based sentiment quadruple (DiaASQ) analysis. By integrating expert-level contrastive learning within a task-oriented mixture of experts (MoE) layer and employing a graph-centric dialogue structuring strategy, TaCoMoE significantly improves the accuracy of extracting sentiment quadruples from complex conversations, especially when considering non-opinion utterances.

What Problem Does This Paper Address?

Extracting structured sentiment quadruples (target, aspect, opinion, sentiment) from multi-turn conversations, known as Dialogue Aspect-based Sentiment Quadruple (DiaASQ) analysis, is a challenging task. Existing approaches struggle with two main issues: accurately extracting multiple elements from complex dialogue structures and effectively leveraging the rich contextual information present in conversations. Furthermore, the impact of non-opinion utterances, which can be prevalent and potentially misleading, on DiaASQ performance has not been thoroughly investigated. This paper aims to address these limitations by proposing a more robust and efficient method for DiaASQ.

Methods

The core of the proposed TaCoMoE framework lies in its novel architecture and training strategy:

  • Task-aware Contrastive Mixture of Experts (TaCoMoE): This approach replaces standard dense layers in LLMs with task-oriented MoE layers. Each expert within the MoE is designed to specialize in different aspects of the task, promoting efficient learning of task-shared and task-specific knowledge.
  • Expert-Level Contrastive Learning: To enhance expert differentiation and representation learning, contrastive learning is incorporated into the MoE layer. Outputs from the same expert are treated as positive pairs, while outputs from different experts are considered negative pairs, encouraging experts to learn distinct features.
  • Graph-Centric Dialogue Structuring (GCDS): To effectively model the complex reply dependencies in dialogues, a GCDS strategy is introduced. This method transforms dialogues into a structured, code-like format, allowing LLMs to better understand the conversational flow without requiring additional graph encoders.
  • Non-opinion Utterance Detection: The framework explicitly incorporates a module to detect and handle non-opinion utterances. This is crucial as these utterances, which do not express sentiment towards a target or aspect, can interfere with accurate quadruple extraction.

Datasets

The experiments were conducted on the DiaASQ dataset, which is described as the first multilingual dataset for dialogue-level aspect-based sentiment analysis. The dataset comprises 1,000 dialogues, available in both Chinese and English, focusing on mobile phone-related topics.

Key Insights and Ideas

  • The Power of Specialized Experts: The MoE architecture, combined with contrastive learning, allows for a more effective division of labor among experts, leading to improved performance by enabling each expert to focus on specific sub-tasks or dialogue characteristics.
  • Dialogue Structure Matters: Representing dialogue reply dependencies using a graph-centric approach (GCDS) significantly aids LLMs in understanding conversational context, outperforming simpler linear textual representations.
  • Don't Ignore the "Non-Opinions": Explicitly identifying and handling non-opinion utterances is not just a minor enhancement but a crucial step for achieving high performance in DiaASQ. Ignoring them leads to a noticeable drop in accuracy.
  • Parameter Efficiency: The use of low-rank matrices within the experts and the overall MoE structure contribute to parameter efficiency, making the method more scalable.

Key Results

  • State-of-the-Art Performance: TaCoMoE consistently outperforms existing state-of-the-art methods on the DiaASQ dataset for both English and Chinese, achieving significant improvements across entity, pair, triplet, and quadruple extraction metrics. For instance, on the English dataset, TaCoMoE achieved a micro F1 score improvement of up to 6.53% over other methods.
  • Impact of Non-opinion Detection: Removing non-opinion detection (w/o NOD) led to a substantial performance decrease, with triplet and quadruple extraction dropping by over 10% and 9% respectively on the English dataset.
  • Effectiveness of GCDS: The GCDS strategy also showed its value, with its removal leading to a noticeable decrease in performance, particularly for cross-utterance quadruples.
  • Optimal Number of Experts: Experiments indicated that setting the number of experts to 8 yielded the best performance.

Limitations and Future Work

The authors acknowledge several limitations and suggest future directions:

  • Training Time: The use of contrastive learning in the MoE layer, treating expert outputs as sample pairs, increases training time.
  • GCDS Generalizability: The effectiveness of the Graph-Centric Dialogue Structuring strategy has not yet been validated on other tasks. While it avoids an additional graph encoder, it can increase context length and memory usage.
  • Non-opinion Nuances: The exploration of non-opinion utterances is preliminary. Further investigation is needed to more effectively distinguish between utterances that truly lack opinions and those where opinions might be implicit or refer to specific targets/aspects.

Summary updated

BibTeX
@inproceedings{he2025task,
  author = {Chenyuan He and Yuxiang Jia and Fei Gao and Senbin Zhu and Hongde Liu and Hongying Zan and Min Peng},
  title = {{Task-aware Contrastive Mixture of Experts for Quadruple Extraction in Conversations with Code-like Replies and Non-opinion Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2025},
}
Figure