Implicit discourse relation recognition means deciding how two adjacent sentences are logically connected (cause, contrast, elaboration, and so on) when the writer left out the connecting word ("because", "however"). It is still hard. Zero-shot ChatGPT reaches 32.97% accuracy on the coarsest four-way PDTB 3.0 task, while supervised systems reach about 70% (NCPrompt). Labeled data is scarce, rare senses collapse into frequent ones, and even human annotators often disagree about which relation holds (Yung et al.). Contrastive learning trains an encoder by pulling "related" pairs of examples together and pushing unrelated ones apart. It has been tried here, but the existing versions define "related" with hand-made rules. Recent theory offers a cleaner view: contrastive learning computes the top eigenvectors of a graph that says which examples count as related (HaoChen et al., Tan et al.). On that view, the design question is no longer which loss to use. It is which graph to build.
We propose to build that graph from connectives, not from text perturbations. A pretrained language model scores how plausible each of roughly a hundred PDTB connectives is when inserted between the two arguments. Two argument pairs are then linked in proportion to how much their plausible connectives overlap. An encoder trained with the spectral contrastive loss on this graph learns a smoothed, low-rank summary of connective substitutability. That summary is the information a discourse relation carries. The model never has to perturb a sentence, which risks flipping its relation, and at test time it needs no connective scoring at all.
Core Claim
The claim has three parts. First, a connective-substitution graph is "well-behaved" in the sense the theory requires: dense within a discourse sense and sparse across senses. Text-augmentation graphs are not, because reordering or negating a clause can turn a Cause into a Result or an Expansion into a Contrast. Second, how much edge weight crosses between gold senses predicts linear-probe quality, as the theory says it should. Third, because the spectral objective keeps only the top-$k$ structure of the graph, it discards the language model's idiosyncratic connective preferences, which act as noise. Directly copying the language model's connective distributions would fit that noise.
The third part is the bet that separates this proposal from simpler alternatives. It rests on Xue et al. and Du et al.. Those papers show that contrastive features occupy the top eigenvectors of the data graph and that noise ends up in the trailing spectrum.
Let $P$ be the matrix whose row $i$ is the language model's distribution $p_i(c)$ over connectives $c$ for argument pair $i$. The graph is bipartite: argument pairs connect to connectives. A positive partner for pair $x$ is drawn by a two-step walk, first a connective and then another argument pair that admits it:
$$ p(x^+ \mid x) = \sum_c p_x(c)\, \frac{p_{x^+}(c)}{\sum_j p_j(c)} . $$
The encoder $f$ is then trained with the spectral contrastive loss of HaoChen et al., where $x'$ is an independent random pair:
$$ \mathcal{L}(f) = -2\,\mathbb{E}_{x,x^+}\!\left[f(x)^\top f(x^+)\right] + \mathbb{E}_{x,x'}\!\left[\left(f(x)^\top f(x')\right)^2\right]. $$
At the population level, the minimizer spans the top-$k$ left singular vectors of a degree-normalized $P$. In other words, the encoder learns a low-rank factorization of connective substitutability that extends to argument pairs it has never seen.
For labeled training data, the rows of $P$ come from human sources instead of the language model: the implicit connectives that PDTB annotators inserted, and the ten-annotator distributions in DiscoGeM.
Relation to Prior Work
Three existing systems touch this space.
Fisher & Simmons applied spectral learning to discourse relations, factorizing moment matrices of a hidden Markov model over discrete features. They reached an F1 of 0.485 with unlabeled New York Times text. Their work motivates ours because it shows that spectral methods plus unlabeled text improve data efficiency. Our proposal differs in that the spectral object is a connective graph and a neural encoder learns its eigenvectors.
GOLF uses soft positive pairs scored by Dice overlap of gold label paths. It is the nearest relative of our approach. However, its graph exists only over labeled examples, and it ties similarity to the fixed sense taxonomy rather than to observed substitutability.
NCPrompt scores multi-token connectives with a next-sentence-prediction head. It treats wrong connectives as hard negatives and pays a multi-pass cost at inference. This conflicts with Yung et al.'s evidence that relations such as Contrast and Concession genuinely co-occur. We use connective scoring only offline, and we let partially overlapping connectives share edge weight.
The soft-graph contrastive methods X-CLR and WSC supply the general technique of contrastive learning on continuous similarity graphs. We reuse it with a graph that is specific to discourse.
Two results constrain the design. The well-behavedness assumption in HaoChen et al.'s guarantees rules out naive text augmentations. I-Ta Lee et al. find that language models favor topical similarity over discourse logic. That is a warning that $P$ itself could be topic-contaminated, so we audit $P$ rather than trust it.
Research Program
The first step is to build $P$. We score the PDTB connective inventory between arguments of every PDTB 2.0, PDTB 3.0, and DiscoGeM instance. We also score several million naturally adjacent sentence pairs that have no explicit connective. Using natively connective-free pairs, rather than explicit relations with the connective deleted, sidesteps the cue-stripping mismatch documented by Lin et al. and Ji et al.. Scoring is done with masked and causal language models of several sizes, so that the noise level in $P$ varies in a controlled way.
The second step is a graph audit. We compute the cross-sense leakage of each candidate graph on gold-labeled data, defined as the fraction of edge weight that joins different Level-2 senses, along with its spectral gap. The candidate graphs are the connective graph, a back-translation graph, a prompt-template graph in the style of NCPrompt, an argument-swap graph, and a sentence-embedding similarity graph in the style of X-CLR. Checking the embedding graph lets us test Lee et al.'s topicality concern directly.
The third step is training. We pretrain a RoBERTa encoder with the loss above on the unlabeled pairs, continue on labeled pairs using human-derived rows, and then attach a classifier. Finally, we analyze which connectives load on each learned eigenvector and whether rare senses such as Expansion.List or Temporal.Synchrony get their own directions.
Baselines
The do-nothing reference is RoBERTa fine-tuned with cross-entropy on gold single labels. It uses the same backbone, data split, and tuning budget as our model. The file reports no number for this exact configuration, so we will run it ourselves. As floors, we also report the majority class and zero-shot ChatGPT, which reaches 32.97% Level-1 accuracy on PDTB 3.0 per NCPrompt.
The strongest discriminative competitors are GOLF and NCPrompt. GOLF (large) reports 69.60% top-level Macro-F1 on PDTB 2.0 (GOLF). NCPrompt reports 70.26% accuracy and 66.73% F1 on PDTB 3.0 Level-1 with an ERNIE backbone (NCPrompt). For fairness we will re-run both on our splits with matched backbone size and an equal hyperparameter search, and report cited and reproduced numbers side by side. Beating them means a higher Level-1 and Level-2 Macro-F1 across five seeds, with the improvement concentrated in minority senses, plus lower inference cost than NCPrompt.
The soft-label competitor is training on annotator distributions as in Yung et al.. No reported numeric result was found on Lacuna for its accuracy or cross-entropy on DiscoGeM. We will therefore run it on DiscoGeM and PDTB 3.0 and compare cross-entropy against the human distributions.
Three ablations isolate the key ingredients. The first is direct distillation, where the encoder regresses onto the rows of $P$ with KL divergence. This uses the same $P$, data, and compute, so any gap is attributable to the spectral, low-rank treatment. The second regresses onto the top-$k$ truncated SVD of $P$, which separates the low-rank target from the contrastive estimator. The third is binary InfoNCE with positives defined as sharing the most likely connective, which removes the soft graph.
Why not just…?
Why not just distill the language model's connective distribution directly? This is the most serious threat. Our method's knowledge also comes entirely from $P$. The spectral argument predicts an advantage only when $P$ is noisy, because a top-$k$ eigenspace discards trailing directions that direct regression would fit (Xue et al.). We test this by varying the size of the scoring model. The prediction is that the gap between the spectral and distillation models grows as the scorer gets smaller and noisier. If distillation ties at every noise level, the contribution is the graph, not the loss, and we will report it that way. If truncated-SVD regression ties with the contrastive loss, the spectral loss is mainly a scalable way to compute that SVD for unseen inputs.
Why not just train on soft labels, as Yung et al. or GOLF do? Soft labels exist only where humans annotated, which covers tens of thousands of pairs. The connective graph extends to millions of unlabeled adjacent pairs. If soft-label fine-tuning matches us when we use 100% of PDTB, our value lies in data efficiency. The learning curves at 10%, 25%, and 50% of the labels will show whether that holds, as an analogue of Fisher & Simmons' finding that semi-supervised training matched full supervision using only half the labels.
Why not just use the language model at inference, by prompt scoring or zero-shot prompting? Zero-shot prompting trails supervised systems by roughly 30 points of Macro-F1 or accuracy (GOLF, NCPrompt). Prompt scoring requires one pass per candidate connective. We include an "argmax of $P$" classifier, which reads the predicted sense straight from the scorer's preferred connective. This tests whether the encoder adds anything beyond the scorer itself.
Why not just use standard augmentation-based contrastive learning? The graph audit answers this directly. If augmentation graphs show leakage across senses that is comparable to or lower than the connective graph, our premise fails.
Decisive Experiments
The theory check comes first. Across the five candidate graphs, the ranking by cross-sense leakage should predict the ranking by linear-probe Level-2 Macro-F1 on PDTB 3.0. A rank reversal would mean the spectral view is not the right lens for this task.
Second, on PDTB 2.0 and PDTB 3.0 at Levels 1 and 2, we compare against the baselines above, separating intra-sentential from inter-sentential relations. Following Liang et al., the two settings differ sharply: they report F1 of about 56.7 and about 35.0 respectively. We expect our gains mainly on the harder inter-sentential case, where unlabeled adjacent pairs best match the test distribution.
Third, we measure cross-entropy against the ten-annotator DiscoGeM distributions, including a test of whether a model trained on Wall Street Journal text transfers across genres. Fourth, we run the noise-scaling comparison against distillation.
The proposal succeeds if the leakage ranking holds and the model beats reproduced GOLF and NCPrompt on Level-2 Macro-F1 and minority-sense F1. It must also beat distillation by a margin that widens as the scorer's noise increases. If only the first two hold, we have a useful graph but no evidence for the spectral mechanism.
References
- NCPrompt: NSP-Based Prompt Learning and Contrastive Learning for Implicit Discourse Relation Recognition
- Label distributions help implicit discourse relation classification
- Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Loss
- Contrastive Learning is Spectral Clustering on Similarity Graph
- Investigating Why Contrastive Learning Benefits Robustness against Label Noise
- When and How Does In-Distribution Label Help Out-of-Distribution Detection?
- Spectral Semi-Supervised Discourse Relation Classification
- Global and Local Hierarchy-aware Contrastive Framework for Implicit Discourse Relation Recognition
- X-Sample Contrastive Loss: Improving Contrastive Learning with Sample Similarity Graphs
- Weakly-Supervised Contrastive Learning for Imprecise Class Labels
- Multi-Relational Script Learning for Discourse Relations
- Recognizing Implicit Discourse Relations in the Penn Discourse Treebank
- Closing the Gap: Domain Adaptation from Explicit to Implicit Discourse Relations
- Extending Implicit Discourse Relation Recognition to the PDTB-3