Wide Parents, Narrow Replies: Gaussian Message Embeddings for Conversation Disentanglement

In a busy group chat, several conversations run at once, and most messages carry no marker of which one they belong to. Disentanglement is the task of recovering those threads. It is usually cast as picking, for each new message, the earlier message it replies to, or declaring that it starts a new thread. Today's best systems score every candidate reply link independently. A telling result shows what this misses. Findings on Conversation Disentanglement reaches 72.6 link F1 on Ubuntu IRC with BERT plus hand-crafted features. When an oracle tells a global matching decoder how many replies each message actually receives, the score jumps to 86.8. So the missing information is largely per-message: how many replies a message will attract, and how sure we should be about where it attaches.

We propose to represent each chat message not as a single vector but as a Gaussian cloud: a center (what the message is about) and a spread (how broadly it can be responded to, or how ambiguous it is). A reply link is scored by an asymmetric divergence, meaning a distance that differs depending on direction, from the reply to its candidate parent. Under this score a wide parent, such as an open question, naturally "covers" many narrow replies, while a terse, specific answer covers almost none. Summing the resulting link probabilities gives each message an expected reply count, which feeds the global decoder in place of the oracle. The same spread also serves as a confidence signal when the training links are noisy.

Core Claim

Gaussian message embeddings scored by asymmetric Kullback-Leibler (KL) divergence make two things fall out of the geometry: how many replies a message receives, and how uncertain its attachment is. Existing point-vector disentanglers must either learn these separately or ignore them. Concretely, for message $i$ with diagonal Gaussian $\mathcal{N}(\mu_i, \mathrm{diag}(\sigma_i^2))$ and an earlier candidate parent $j$, the link energy is

$$E(i \to j) = \tfrac{1}{2}\sum_{d}\Big[\tfrac{\sigma_{i,d}^2}{\sigma_{j,d}^2} + \tfrac{(\mu_{j,d}-\mu_{i,d})^2}{\sigma_{j,d}^2} - 1 + \log\tfrac{\sigma_{j,d}^2}{\sigma_{i,d}^2}\Big],$$

where $d$ indexes embedding dimensions. This energy is small when the child's cloud sits inside a broad parent's cloud and large in the reverse direction. We add a learned broad "new thread" Gaussian as an extra candidate, so the start-of-thread decision becomes a comparison against a prior rather than a separately thresholded self-link. The claim is testable in three ways. The expected degree $\hat{k}_j = \sum_i p(i \to j)$, where $p$ is a softmax over $-E$ within each child's candidate set, should close a meaningful share of the 72.6-to-86.8 oracle gap when fed to bipartite matching. Spread should predict link errors better than confidence from deterministic scores. And the model should degrade more gracefully when trained on corrupted reply links.

Why This Intersection, and What Is New

Three pieces of probabilistic node representation work supply the method. Deep Gaussian Embedding of Graphs (Graph2Gauss) maps nodes to Gaussians from their attributes and trains with KL-based ranking. It showed that learned variance tracks neighborhood diversity: bridge nodes and hubs get wide clouds. Low-dimensional statistical manifold embedding of directed graphs showed that asymmetric KL encodes directed reachability without separate source and target vectors. That property is exactly what the reply-to relation needs. Robust Attribute and Structure Preserving Graph Embedding (RASE) showed that Gaussian embeddings lose only 3.8% on CiteSeer under 50% injected edge noise, where Graph2Gauss dropped more than 14%. RASE uses the symmetric 2-Wasserstein distance instead of KL, which creates a design tension we test directly.

The conversation side motivates the problem. Kummerfeld et al.'s A Large-Scale Corpus for Conversation Disentanglement framed disentanglement as reply-link prediction. They reported that heuristically built dialogue graphs are about 80% corrupted, so training on noisy links is the realistic regime. Online Conversation Disentanglement with Pointer Networks identifies the start-of-thread decision as fragile. Conversation- and Tree-Structure Losses for Dialogue Disentanglement (DIALBERT) degrades on distant parents.

The closest existing uses of Gaussians in this area are different in kind. Revisiting Conversation Discourse for Dialogue Disentanglement uses a Gaussian only as a fixed scalar edge weight over turn distance, and its message vectors remain deterministic. Variational Graph Recurrent Neural Networks and Hyperbolic Variational Graph Neural Network put distributions on users in interaction networks, whose node sets are fixed or slowly changing. None represents messages as distributions, and none ties spread to reply count or to the decoder. We deliberately do not pursue emotion recognition or reading comprehension here. The disentanglement setting has a clean oracle gap that makes the hypothesis falsifiable.

Research Program

The encoder matches Zhu et al.'s BERT+MF setup for fairness: BERT over the message with local context, concatenated with the same timestamp, speaker and mention features. The difference is the output. Two heads produce $\mu_i$ and $\log\sigma_i^2$, with the variance lower-bounded to avoid collapse. Training minimizes the negative log-likelihood of the gold parent under the softmax over $-E$, within a candidate window. KL between diagonal Gaussians is cheap, so we can afford windows longer than the 50-message pointer-network convention. We test windows of 50, 100 and 200 to probe the distant-parent failure DIALBERT reports.

A second, optional loss asks each message's predicted expected degree to match its observed child count. Toggling this loss separates what the geometry gives for free from what supervision adds.

Decoding has three variants, run on identical scores. The first is greedy argmax. The second is Zhu et al.'s maximum-weight bipartite matching with capacities set to rounded $\hat{k}_j$. The third is the same matching with oracle capacities, which serves as a ceiling.

Graph2Gauss and the file's tension note both warn that unimodal Gaussians may be too weak for messages that straddle two threads. We therefore also build a two-component mixture variant. We adopt it only if it helps specifically on messages annotated with multiple parents.

Baselines

The naive reference point is Kummerfeld et al.'s feedforward model, at 36.2 exact-match thread F1. The strongest link-level competitors are the pointer network of Yu and Joty, which reports 73.1 link F1 and 80.1 on the one-to-one clustering measure, and Zhu et al.'s BERT+MF at 72.6 link F1. At the clustering level, the strongest competitors are DIALBERT, at 76.3 ARI and 93.9 on variation of information (VI) in the file's reporting, and the heterogeneous discourse graph of Bobo Li et al. That paper reports a +7.50 ARI gain over its prior best baseline and 0.72 link F1. We treat it as the strongest existing method. It is also the closest in spirit, because it already builds a relational graph over messages.

For fairness, every learned baseline we re-run uses the same BERT backbone, the same metadata features, the same candidate window and the same matching decoder where applicable. Each gets an equal hyperparameter search budget: the same number of trials over learning rate and window size, on the same development split. Beating the field means exceeding 73.1 link F1 and 76.3 ARI on the standard Ubuntu IRC test split, averaged over five seeds and with non-overlapping confidence intervals. It also means recovering a stated share of the oracle gap: at least a third of the distance from 72.6 toward 86.8 when matching uses predicted capacities.

The ablation that isolates the key ingredient is a deterministic twin. It uses the same encoder and a bilinear link score, plus a separate regression head that predicts each message's reply count, fed to the same matching decoder. A second ablation replaces KL with 2-Wasserstein distance, keeping Gaussians but removing asymmetry. A third fixes all variances to a constant, which reduces the energy to a scaled Euclidean distance.

Why not just…?

Why not just add a reply-count head to a deterministic model? This is the most serious threat. If the deterministic twin with a supervised degree head matches our link F1 and oracle-gap recovery, then the Gaussian geometry adds nothing to the decoding claim, and we will report that. Our bet is on two advantages. KL coverage couples degree and link choice in a single score, so degree estimates stay consistent with the link probabilities rather than being learned in isolation. And spread additionally gives an uncertainty signal and noise robustness that a scalar count head does not. The noise and calibration experiments below test those advantages separately from raw F1.

Why not just prompt a large language model with the chat log? Revisiting Conversation Discourse for Dialogue Disentanglement reports GPT-3.5 zero-shot at 0.10 link F1, against 0.72 for a trained graph model. We will include an LLM zero-shot row for reference, but the file gives no evidence that prompting is competitive on this task.

Why not just use Monte Carlo dropout or an ensemble on a deterministic scorer for uncertainty? Both give calibrated-looking link probabilities without Gaussian nodes. They cost several forward passes, though, and they express uncertainty about links rather than a per-message property that feeds the decoder. Learning Latent Graph Structures and their Uncertainty further shows that point-prediction training can settle on wrong structures that still predict well, leaving uncertainty miscalibrated. We will run a five-member ensemble of the deterministic twin and compare it on error-detection AUROC and calibration at matched compute.

Why not just rely on metadata features? Timestamps and mentions already carry most of the gain on IRC. Yu and Joty's link F1 falls from about 73 to 24 without speaker and time features. We keep those features in every model. We also evaluate on Movie Dialogue and Reddit-large, where metadata is weaker, to see whether spread compensates.

Proposed Experiments

The main benchmark is the Ubuntu IRC corpus, reporting link F1, exact-match thread F1, ARI, VI, one-to-one overlap and self-link F1. Transfer is tested on Reddit-large, where DIALBERT reports 36.1% graph accuracy against 29.6% for BERT, and on Movie Dialogue. There, Bi-CL (Conversation Disentanglement with Bi-Level Contrastive Learning) reports 0.747 Shen-F. Bi-CL's hard K-means prototypes also make it a natural contrast for soft Gaussian membership.

The first decisive check is the oracle-gap experiment: matching with predicted versus oracle capacities, against the deterministic degree-head twin. We also report the correlation between $\hat{k}_j$ and true reply counts.

The second is a label-noise sweep. We corrupt 0, 10, 30 and 50% of training reply links, by random reassignment and by a timestamp-proximity heuristic that mimics real parsers, and plot test link F1 and ARI for every model. The prediction is a flatter curve for the Gaussian model, echoing RASE's result on citation graphs.

The third is uncertainty quality. We measure how well mean spread identifies wrongly attached messages (AUROC). We also measure selective accuracy when abstaining on the highest-spread 10% of messages, compared with softmax margin and with the ensemble.

The fourth is interpretability. We test whether spread is systematically larger for short acknowledgments, for messages with multiple annotated parents, and for thread-opening questions, as Graph2Gauss's diversity finding predicts.

The fifth is the KL-versus-Wasserstein and window-length ablations, reported separately on links spanning more than 50 messages.

A negative result on the degree-head comparison, combined with a positive one on noise robustness, would still be informative. It would narrow the contribution to robust training, and we will report it that way.