Parsing a group chat means recovering its discourse structure. The parser decides, for each message, which earlier message it responds to (the "link") and what kind of response it is: an answer, an elaboration, a contrast, a piece of background (the "relation"). Current parsers are good at the first decision and at common relation types. They fail on rare relations, the subtle rhetorical moves that make up the tail of the label distribution. Two recent papers that report strong results on the standard benchmarks both name this failure. The authors of the Molweni dataset single out relations like Alternation and Parallel as the key open problem. The explanation-distillation parser DDPE still struggles with Background and Narration despite distilling rationales from GPT-4 Turbo.
We propose a simple change to how these parsers are trained. Today the two decisions are trained as separate tasks with separate losses. Instead, we apply a single difficulty-based reweighting, focal loss (a loss that shrinks the contribution of examples the model already gets right), to the probability of the whole correct answer: the right parent and the right relation together. Because this joint probability is a product of the link probability and the relation probability, one weight ends up governing both tasks for each message. A rare relation sitting on an easy link stays "hard" and keeps its gradient. The model does not need hand-set task weights to keep the link loss from drowning out the relation loss. We also test whether the same focusing, applied to a noisy auxiliary task, can replace the reinforcement-learning data selector that prior multi-task parsers needed.
Core Claim
In hierarchical discourse parsing, label imbalance lives at two coupled levels: which earlier utterance is the parent, and which of the 17 relation types from SDRT (Segmented Discourse Representation Theory, the annotation scheme used by Molweni and STAC) labels the edge. Focusing on joint difficulty should help the rare relations more than focusing on each level separately. It should also make most of the explicit balancing between the two tasks unnecessary.
Let $j$ be a dependent utterance with gold parent $h^*$ and gold relation $r^*$. Let $p_\ell = p(h^* \mid j)$ be the model's probability of the gold parent, from a softmax over earlier utterances. Let $p_r = p(r^* \mid j, h^*)$ be the probability of the gold relation given that parent. The standard multi-task objective, with or without per-head focal loss, treats the two terms separately:
$$\mathcal{L}_{\text{indep}} = -(1-p_\ell)^{\gamma}\log p_\ell \;-\; \lambda\,(1-p_r)^{\gamma}\log p_r .$$
Here $\gamma \ge 0$ is the focusing exponent and $\lambda$ is a task weight that must be tuned. Our objective focuses on the joint gold probability $q = p_\ell\, p_r$:
$$\mathcal{L}_{\text{joint}} = -(1-q)^{\gamma}\,\big(\log p_\ell + \log p_r\big).$$
The weight $(1-q)^\gamma$ is shared by both terms. When the link is easy but the relation is rare and misclassified, $q$ is small, so the link term keeps its weight and the relation term is not suppressed. When both are easy, both vanish together.
The plain focal loss of Lin et al. is the per-head special case. In the global tree model of Chi and Rudnicky, $q$ is replaced by the Matrix-Tree marginal of the gold labeled edge. The Matrix-Tree Theorem lets that model compute exact probabilities for each edge under a distribution over whole spanning trees.
We are explicitly not claiming that link prediction suffers from crippling quadratic negative imbalance. The research file frames it that way, but most strong parsers already pick each utterance's parent with a softmax over earlier utterances. That leaves $O(N)$ candidates per dependent, normalized together, so the "millions of easy negatives" regime that motivated focal loss in object detection largely does not apply. The real starvation is in the relation tail and in how the link task and relation task share gradient. That is where we put the mechanism.
Research Program
The work has three parts. First, we implement $\mathcal{L}_{\text{joint}}$ on two backbones that represent the two sides of a known tension in the literature. One is a local pairwise scorer in the style of DialogDP, which scores every utterance pair independently with biaffine attention, uses a soft distance mask, and allows multi-parent structures. The other is the global spanning-tree parser of Chi and Rudnicky. Both build on a pretrained encoder with utterance-level structure, following the two-level design that HRED introduced. Using both backbones tests whether the effect depends on how the structure is normalized.
Second, we study calibration. Focal loss is known to distort output probabilities, a problem analyzed in Functional Properties of the Focal-Entropy and addressed by AdaFocal and Dual Focal Loss. Distorted probabilities matter here because joint decoding adds link and relation log-scores together, and tree decoding with the Chiu-Liu-Edmonds algorithm compares these scores across edges. We measure relation-level expected calibration error with and without temperature scaling fit on the dev set. We also test whether recalibrating before decoding changes Link&Rel F1.
Third, we add the auxiliary reply-to task. The Task-aware Structure Transformer (TST) pairs discourse parsing with addressee recognition (predicting who replied to whom), trained on 1.75 million auxiliary dialogues. It needed an actor-critic selector that discarded roughly 80% of that data to avoid negative transfer. Our hypothesis is that much of the discarded data is easy (obvious reply-to links). Applying the joint focal weight to the auxiliary examples would then down-weight them automatically, acting as a soft, free data selector. We will measure this directly by correlating per-example focal weights with an offline estimate of each example's usefulness.
Baselines
The naive reference point is the same backbone trained with plain cross-entropy on both heads and a tuned $\lambda$. On Molweni, the Molweni paper reports that its Deep Sequential baseline reaches 78.1 link F1 and 54.8 Link&Rel F1. Our reimplemented cross-entropy backbones should sit well above this. We will report them as the controlled zero point for every comparison.
The ablation that isolates our key ingredient is per-head focal loss, $\mathcal{L}_{\text{indep}}$, with $\gamma$ tuned separately for each head and $\lambda$ tuned as well. If joint focusing does not beat this on tail-relation F1, the coupling idea is wrong. A second ablation uses a "detached" joint weight, with $(1-q)^\gamma$ applied only to the relation term. This tells us whether the benefit comes from the shared weight acting on the link term or only from the relation side.
The strongest generic competitor is the classic fix for a long-tailed label set: class-frequency-weighted cross-entropy (the $\alpha$ term of $\alpha$-balanced focal loss used alone) on the relation head, combined with learned multi-task balancing. For the balancing we use the bilevel scheme of Scalable Bilevel Loss Balancing for Multi-Task Learning, or uncertainty-style task weights if reimplementing that scheme proves infeasible. This baseline directly targets both problems we claim to address, without any coupling.
The strongest published systems are DP-DCM-CPO from Discourse-aware Utterance Clarification, which reports Link&Rel F1 of 66.2 on Molweni and 69.0 on STAC, and DDPE, which reports 87.6 link F1 and 62.9 Link&Rel F1 on Molweni and 79.5 and 63.4 on STAC. TST reports 85.26 link and 60.91 Link&Rel F1 on Molweni and 73.69 link F1 on STAC. DP-DCM-CPO changes the input by rewriting utterances that omit information. DDPE uses a LLaMA-3 student and GPT-4 Turbo rationales. Neither is a loss-level competitor, and neither is compute-matched to our backbones.
We handle this in two ways. All loss variants share the same encoder, data, seeds (five per configuration), and an equal hyperparameter-search budget of the same number of dev-set trials for each method's free parameters. Separately, we apply $\mathcal{L}_{\text{joint}}$ to the clarified inputs that DP-DCM-CPO's approach produces, if they can be regenerated, to test whether the two improvements stack.
"Beating" the controlled baselines means a gain in macro-averaged relation F1 and in the mean F1 of the five rarest relations that exceeds seed variance, on both Molweni and STAC, with no loss in link F1. The file contains no per-relation numbers: no reported result found on Lacuna for per-relation or macro relation F1 on Molweni or STAC. We will therefore produce these numbers ourselves for every baseline. Matching or exceeding 66.2 Link&Rel F1 on Molweni is a stretch goal, not the success criterion.
Why not just…?
Why not just reweight rare relation classes in cross-entropy? This is the most serious threat. Frequency weighting is cheap and directly targets the tail. It may well recover most of the gain. The difference is that frequency weights are fixed per class. They over-weight easy instances of rare classes, which hurts precision, and they never respond to link difficulty. The class-weighted plus bilevel-balancing baseline is run at equal tuning budget precisely to settle this. If it matches $\mathcal{L}_{\text{joint}}$ on tail F1 and calibration, we will report that the coupling adds nothing.
Why not just apply focal loss separately to each head? That is $\mathcal{L}_{\text{indep}}$, and it is our main ablation. The argument against it is mechanical. Per-head focusing leaves the link term's weight dependent only on link difficulty, so a dependent whose relation is wrong but whose parent is easy contributes almost nothing to the shared encoder through the link path. Still, it requires tuning $\lambda$. If tuning $\lambda$ well closes the gap, the coupling is only a convenience, and we will say so.
Why not just distill explanations from a large language model, as DDPE does? DDPE's own authors report that this still fails on Background and Narration. That suggests extra rationales do not fix gradient starvation for classes with few examples. The approaches are complementary rather than exclusive. We test the loss on a compute-matched encoder, and if resources allow, we also test it on a LLaMA-scale student. This shows whether the tail gain survives at that scale.
Why not just keep the reinforcement-learning data selector for the auxiliary task? TST shows the selector works. But it requires an actor-critic loop with validation rewards, which costs far more than a reweighting term. Experiment three compares the two head-to-head on the same auxiliary data. If focal down-weighting falls clearly short of the selector, we conclude that the useless auxiliary examples are noisy rather than easy. That would be a useful negative result.
Proposed Experiments
The primary benchmarks are Molweni and STAC. We report link F1, Link&Rel F1, macro relation F1, per-relation F1 for Background, Narration, Alternation, Parallel, and Result, F1 broken down by link distance (since links spanning four or more turns are a known weak point), and relation expected calibration error.
The first decisive check is the head-to-head among cross-entropy, class-weighted with balancing, $\mathcal{L}_{\text{indep}}$, detached-joint, and $\mathcal{L}_{\text{joint}}$ on both backbones. The prediction is that $\mathcal{L}_{\text{joint}}$ improves tail-relation F1 most on the local biaffine backbone. There, link and relation are scored independently, so coupling them should matter more. We expect a smaller gain on the tree backbone, whose marginals already couple edges globally.
The second check sweeps $\gamma$ with $\lambda$ fixed at 1. The prediction is that $\mathcal{L}_{\text{joint}}$ is far less sensitive to $\lambda$ than $\mathcal{L}_{\text{indep}}$, because the coupling should do most of the balancing.
The third check adds addressee-recognition data under four conditions: unfiltered with cross-entropy, unfiltered with $\mathcal{L}_{\text{joint}}$, a random 20% subset, and a selector-filtered subset if one can be obtained. It also measures the correlation between per-example focal weight and each example's estimated benefit.
As a transfer check, we train a disentanglement link head on Ubuntu IRC with the same loss. That setting has only links and no relations, so the coupling should give no advantage over per-head focusing. A gain there would mean our explanation of where the benefit comes from is incomplete.
References
- Molweni: A Challenge Multiparty Dialogues-based Machine Reading Comprehension Dataset with Discourse Structure
- Enhancing Multi-party Dialogue Discourse Parsing with Explanation Generation
- Focal Loss for Dense Object Detection
- Structured Dialogue Discourse Parsing
- Task-Aware Self-Supervised Framework for Dialogue Discourse Parsing
- Building End-To-End Dialogue Systems Using Generative Hierarchical Neural Network Models
- Functional Properties of the Focal-Entropy
- AdaFocal: Calibration-aware Adaptive Focal Loss
- Dual Focal Loss for Calibration
- Improving Dialogue Discourse Parsing via Reply-to Structures of Addressee Recognition
- LDC-MTL: Balancing Multi-Task Learning through Scalable Loss Discrepancy Control
- Improving Dialogue Discourse Parsing through Discourse-aware Utterance Clarification