Introduction
In modern digital communication platforms like Slack, IRC, or WhatsApp, a single channel often hosts multiple simultaneous conversations. This results in a "jumbled" stream of messages where distinct threads are intermingled, making it difficult for both human users and automated systems to follow specific dialogue flows. The task of conversation disentanglement aims to solve this by clustering messages into their original, separate threads.
The Disentanglement Problem
Traditional approaches to this problem typically treat it as a two-step supervised learning task. First, a model predicts local "reply-to" links (identifying which previous message a new message is responding to). Second, these individual links are grouped into global clusters or trees. However, a significant gap exists between training and evaluation: while models are trained to optimize local link accuracy, they are evaluated on global metrics like the Adjusted Rand Index (ARI) or Variation of Information (VI). These global metrics are non-differentiable and cannot be directly optimized using standard supervised learning techniques.
End-to-End Reinforcement Learning Framework
To bridge this gap, this paper proposes the first end-to-end Reinforcement Learning (RL) approach for conversation disentanglement. By framing the task as a sequential decision-making process, the model can directly optimize for the global conversation structure.
Policy Network
The underlying policy uses Structural BERT, which generates a probability distribution over potential "parent" messages within a fixed time window. This network combines context-aware BERT representations with a Relational Graph Convolutional Network (r-GCN) to capture complex speaker relationships and dialogue history.
The RL Training Loop
Using the REINFORCE algorithm, the agent samples trajectories of linking decisions. These decisions form a predicted cluster, which is then evaluated by a reward function. The resulting gradients are used to update the policy network, allowing it to "see" the long-term impact of individual message links on the final conversation structure.
Reward Innovation: micro TL-FBC
A key finding of this research is that standard global metrics (like ARI) can provide "spurious signals" when used as RL rewards. An agent might receive a high reward for a cluster that contains the correct messages but possesses a completely incorrect internal link structure. This is problematic because conversation threads are not just sets of messages; they are structured trees where root links are more critical than leaf links.
To address this, the authors introduced the micro Thread-Level FBcubed (micro TL-FBC) reward function. This metric treats a conversation as a "bag of threads" (paths from root to leaf) rather than a "bag of messages." By assigning higher weights to links at the top of a conversation tree, the reward specifically penalizes "cascading errors"—where a single wrong link at the root level causes an entire sub-tree to be misclassified.
Key Results
The RL-based model was evaluated on the Ubuntu IRC dataset, a gold-standard benchmark for technical support chat disentanglement. The results showed significant improvements over previous state-of-the-art models:
- Link-Level Performance: Achieved an F1 score of 83.3, compared to 75.4 for the baseline Structural BERT.
- Global Structure: Improved the ARI to 78.5 (from 68.4) and the Variation of Information (VI) to 96.2 (from 93.0).
- Self-Link Accuracy: The model demonstrated a superior ability to identify messages that start new threads, which is a critical factor in preventing cascading errors.
Conclusion
By moving from local link prediction to end-to-end global optimization, this RL approach effectively bridges the gap between training objectives and evaluation metrics. The introduction of the micro TL-FBC reward ensures that the model learns the structural nuances of dialogue, leading to more coherent and accurate conversation disentanglement in complex, multi-user environments.
Summary updated