When we read the sentences "It started raining. I opened my umbrella," we instantly perceive a causal link. Even though the word "so" or "because" is missing, the logical relationship is clear. For machines, however, identifying these implicit discourse relations is a notoriously difficult task. Without explicit markers like "but," "however," or "since," models often struggle to grasp the underlying logic connecting two spans of text.
Recent research has converged on a powerful intuition: if a model can't find a connective, we should ask it to invent one. By transforming discourse analysis into a "fill-in-the-blank" task, researchers are leveraging the latent linguistic knowledge within pre-trained language models to make the implicit explicit.
The Cloze Task as a Logical Probe
The core idea behind this direction is to treat discourse relation recognition not as a standard classification problem, but as a cloze-style task. In a typical setup, a model is presented with two text segments separated by a mask token: [Segment 1] [MASK] [Segment 2].
As Murathan Kurfalı and Robert Östling demonstrate in Let’s be explicit about that: Distant supervision for implicit discourse relation classification via connective prediction, a pre-trained language model can rank candidate connectives (like "but" vs. "because") based on how well they fit the context. This approach works because models like BERT or RoBERTa were originally trained to predict missing words. By placing the mask at the junction of two sentences, researchers "probe" the model's internal understanding of the transition.
Aligning Tasks with Pre-training
Standard classification often requires adding a new, randomly initialized "head" to a model, which then needs to be trained from scratch. This creates a gap between how the model was pre-trained (predicting words) and how it is being used (predicting labels like Comparison or Contingency).
To bridge this gap, Wei Xiang and colleagues introduced TEPrompt in TEPrompt: Task Enlightenment Prompt Learning for Implicit Discourse Relation Recognition. By using natural language templates, they align the downstream task with the model's original training objective. Similarly, Chenxu Wang and Ping Jian developed the PLSE method, which uses prompt-based learning to enhance logical semantics. The intuition here is that the model is already an expert at predicting connectives; by framing the problem this way, we are simply asking the model to speak its native language.
From Prediction to Classification
Predicting a connective is often just the first step. The ultimate goal is usually to categorize the relation into a formal taxonomy (e.g., the Penn Discourse Treebank). Researchers have developed several ways to move from a predicted word to a final label:
- Direct Mapping: Predicted tokens (like "specifically") are mapped directly to categories (like
Expansion.Restatement). Subhabrata Dutta and Tanmoy Chakraborty use this in Can Unsupervised Knowledge Transfer from Social Discussions Help Argument Mining? to identify claims and premises in complex threads. - Two-Stage Generation: Wei Liu and Michael Strube propose a model in Annotation-Inspired Implicit Discourse Relation Classification with Auxiliary Discourse Connective Generation that mimics human annotators. The model first generates a "missing" connective and then uses that generated word as an additional input feature for a final classifier.
- Knowledge Distillation: In Connective Prediction for Implicit Discourse Relation Recognition via Knowledge Distillation, Hongyi Wu and colleagues use a "teacher" model to provide soft, nuanced labels for connectives, helping a "student" model learn the subtle differences between similar logical links.
Expanding the Horizon
While much of this work began on news corpora, the field is moving toward more diverse data. Yang Janet Liu and the team behind GDTB: Genre Diverse Data for English Shallow Discourse Parsing across Modalities, Text Types, and Domains have shown that while connective prediction is a robust signal, its effectiveness can vary across different genres, from social media to dialogue.
In the context of dialogue, Gongyao Jiang and Meishan Zhang demonstrate in A Pilot Study on Dialogue-Level Dependency Parsing for Chinese that these signal-based methods can even help transform syntactic trees into discourse structures, proving that the "connective-as-a-signal" approach is a versatile tool for uncovering the hidden architecture of human language.