Conversation Disentanglement with Bi-Level Contrastive Learning

Chengyu Huang, Zheng Zhang, Hao Fei, Lizi Liao

Conversation disentanglement is the challenging task of partitioning a multi-party chat log into separate, coherent threads or "sessions." In environments like IRC, Slack, or Discord, multiple topics often overlap, making it difficult for automated systems to follow a single line of reasoning. This paper, presented at EMNLP 2022, introduces Bi-CL (Bi-Level Contrastive Learning), a framework designed to address these complexities by modeling conversations at both the individual utterance and global session levels.

The Challenge of Entangled Conversations

Existing disentanglement methods typically rely on a two-step "pairwise" approach: first predicting if one message is a reply to another, and then clustering these pairs into threads. However, this approach suffers from two primary limitations:

  1. Local Myopia: By focusing solely on pairs, models often miss the broader semantic context of a session.
  2. Data Scarcity: Pairwise models require extensive human-annotated "reply-to" labels, which are costly and time-consuming to produce.

Methodology: Bi-Level Contrastive Learning

The proposed Bi-CL framework utilizes a hierarchical encoding structure to capture both local and global dependencies.

Architecture of the Bi-CL framework showing hierarchical encoding and bi-level contrastive learning objectives.

The core of the model lies in its dual contrastive objectives:

  • Utterance-Level Contrastive Loss ($L_u$): This objective pulls representations of messages belonging to the same session closer together in the embedding space while pushing messages from different sessions apart.
  • Session-Level Contrastive Loss ($L_s$): This "bi-level" innovation maintains "prototypes" (centroids) for each session. It encourages individual messages to align with their respective session prototypes, effectively capturing the global semantic "vibe" of the thread.

For unsupervised scenarios, Bi-CL employs an Expectation-Maximization (EM) framework. It iteratively clusters messages using K-Means to identify "pseudo-prototypes" (E-step) and then updates the model to pull messages toward these centers (M-step). Additionally, a sub-network predicts the number of distinct threads ($K$) within a text block to guide the clustering process.

Evaluation and Results

The model was evaluated on two benchmark datasets: Ubuntu IRC, which contains complex technical support logs with high concurrency, and Movie Dialogue, a more controlled dataset of intermingled movie script lines.

Performance comparison of Bi-CL against supervised and unsupervised baselines on Ubuntu IRC and Movie Dialogue datasets.

As shown in the results table, Bi-CL achieved state-of-the-art (SOTA) performance across multiple metrics, including Normalized Mutual Information (NMI), Adjusted Rand Index (ARI), and the Shen-F score. Notably:

  • In the supervised setting on Movie Dialogue, Bi-CL reached a Shen-F score of 0.747, significantly surpassing previous models like StructBERT.
  • In the unsupervised category, Bi-CL dominated the Ubuntu IRC benchmark, achieving an ARI of 0.345, nearly doubling the performance of the "Co-Training" baseline.

Ablation studies confirmed that the session-level loss ($L_s$) is critical for performance, as its removal led to a significant drop in clustering quality.

Conclusion and Future Directions

Bi-CL demonstrates that modeling global session structures through contrastive learning is a powerful alternative to traditional pairwise methods. The small performance gap between supervised and unsupervised versions suggests that the inherent semantic structure of conversation threads is highly informative.

Future work may involve integrating this bi-level loss with larger Transformer-based architectures like BERT or GPT and addressing the challenges of handling extremely long-range dependencies in massive chat logs.

Summary updated

Abstract

Conversation disentanglement aims to group utterances into detached sessions, which is a fundamental task in processing multi-party conversations. Existing methods have two main drawbacks. First, they overemphasize pairwise utterance relations but pay inadequate attention to the utterance-to-context relation modeling. Second, huge amount of human annotated data is required for training, which is expensive to obtain in practice. To address these issues, we propose a general disentangle model based on bi-level contrastive learning. It brings closer utterances in the same session while encourages each utterance to be near its clustered session prototypes in the representation space. Unlike existing approaches, our disentangle model works in both supervised setting with labeled data and unsupervised setting when no such data is available. The proposed method achieves new state-of-the-art performance on both settings across several public datasets.
BibTeX
@inproceedings{huang2022conversation,
  author = {Chengyu Huang and Zheng Zhang and Hao Fei and Lizi Liao},
  title = {{Conversation Disentanglement with Bi-Level Contrastive Learning}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2022},
  doi = {10.18653/v1/2022.findings-emnlp.217},
}
Figure