Self-Supervised Logical Reasoning via Discourse Marker Prediction

49 concepts35 papers

When we read, the meaning of a text emerges not just from individual words, but from how sentences connect to form a coherent narrative. Consider these two sentence pairs:

  1. The roads were icy because it froze overnight.
  2. The roads were icy. It froze overnight.

In the first pair, the relationship is explicit, signaled by the discourse marker because. In the second pair, the relationship is implicit. As human readers, we instantly infer the same causal link in both cases. For machine learning models, however, recognizing implicit discourse relations is notoriously difficult. Training a model to understand these unstated logical links typically requires massive amounts of human-annotated data, which is expensive and difficult to scale.

To bypass this bottleneck, a powerful paradigm has emerged: using explicit discourse markers as a free, automated source of supervision to teach models the underlying logic of natural language.

Connectives as Free Semantic Labels

The core insight of this research direction is that raw, unannotated text contains millions of explicit discourse markers (but, because, however, therefore) that act as naturally occurring logical labels. By automatically mining these sentences and stripping out the connectives, researchers can create massive self-supervised training datasets.

As demonstrated by Nie et al. with the DisSent framework, and similarly by Sileo et al., a model can be trained on a simple proxy task: given two text segments, predict the specific discourse marker that originally connected them.

Why does this simple task work? To correctly guess whether the missing word is because or although, the model cannot rely on superficial token matching. It is forced to analyze the semantic content of both clauses and determine their rhetorical relationship (e.g., causality vs. contrast). Zhang et al. used perturbation methods to show that optimizing for discourse relation prediction forces sentence encoders to prioritize core asserted semantic content over peripheral details or human annotation biases, yielding high-quality, robust sentence representations.

[Segment 1: The stock price plummeted]  ____  [Segment 2: profits hit a record high]
                                         │
                                 ┌───────┴───────┐
                                 ▼               ▼
                        Predict Connective: "although"
                                 │
                                 ▼
            Forces model to learn the concept of CONTRAST

Shifting the Masking Paradigm from Words to Structure

Standard language model pre-training (like BERT's Masked Language Modeling) masks random tokens, which often forces the model to focus on localized factual or lexical associations. To teach models structural logic, researchers have adapted this masking strategy to target the "glue" of language.

For instance, Dutta et al. introduced Selective Masked Language Modeling (sMLM), which specifically targets discourse markers and logical connectives during pre-training rather than random words. Similarly, the APOLLO framework proposed by Sanyal et al. targets conjunctions and adverbs in logically rich text. By forcing the model to fill in these structural blanks, the training objective shifts the model's attention away from noun-based factual knowledge and toward the logical dependencies between entire clauses.

This structural focus can also be applied at different granularities. Zhang et al. demonstrated that a second-stage pre-training process focusing on Discourse Marker Prediction at the Elementary Discourse Unit (EDU) level significantly improves transition-based discourse parsing by embedding rhetorical logic directly into the model's internal representations.

Bridging the Explicit-to-Implicit Gap

The ultimate goal of learning from explicit markers is to transfer that knowledge to text where no markers exist. Ji et al. originally framed this transition as a domain adaptation problem, developing techniques to handle the systematic linguistic differences between text written with connectives versus text written without them.

To make this transfer successful, modern architectures often employ multi-task learning or shared latent spaces:

  • Latent Sense Spaces: Ru et al. showed that mapping sentence pairs to a latent sense space before predicting discourse markers allows the model to capture ambiguous or entangled relations, making the learned features highly transferable to implicit contexts.
  • Joint Objectives: Single-author work by Bai, as well as work by Meng et al., utilized auxiliary objectives where a model simultaneously predicts an abstract discourse relation and the most likely missing connective word. This grounds abstract logical categories in concrete lexical markers.
  • Unified Latent Spaces: The DISARM framework by Contalbo et al. uses multi-task and adversarial training to align abstract argument relations (like "support" or "attack") with the linguistic patterns of explicit discourse markers into a single representation space.

This explicit-to-implicit transfer is powerful enough to cross task boundaries. For example, Shi et al. demonstrated that discourse relations learned within single sentences (intra-sentence) successfully transfer to classifying implicit relations that span across separate sentences (inter-sentence).

Downstream Generalization: From Discourse to Reasoning

Why does discourse marker prediction improve broader natural language understanding? Because the rhetorical patterns signaled by connectives serve as a foundational scaffolding for high-level reasoning.

Natural Language Inference and Stance Classification

In Natural Language Inference (NLI) and stance detection, a model must determine if a premise entails or contradicts a hypothesis. Pan et al. developed the Discourse Marker Augmented Network (DMAN), which uses reinforcement learning to transfer logical common sense from discourse markers to NLI benchmarks.

Similarly, Jin et al. used discourse connectives as "silver labels" to train stance classification models on massive unlabeled corpora. Because the model learns the abstract structure of argumentation (e.g., how but signals an opposing stance) rather than topic-specific keywords, it generalizes far better to unseen topics. This distant supervision approach has even been shown by Shaik et al. to help specialized models outperform massive large language models on complex scientific NLI benchmarks.

Narrative and Event Reasoning

Discourse structure is intimately tied to how stories and timelines are constructed. Tang et al. showed that knowledge from discourse relation recognition can be projected onto event relation extraction, helping models identify when one event causes or precedes another even when explicit timeline markers are missing. By explicitly embedding these relations, the CoCoLM model developed by Yu et al. achieved significant performance gains on multi-hop narrative and causal reasoning tasks.

Pragmatics and Document Flow

Finally, understanding discourse markers helps models grasp the broader pragmatic constraints of language. Nadeem et al. demonstrated that pre-training on discourse marker prediction provides a structural "warm start" that allows neural models to evaluate the logical flow and organization of essays.

At a deeper linguistic level, Kabbara et al. found that pre-finetuning models on discourse tasks serves as an effective proxy for learning pragmatic constraints, such as identifying presupposition satisfaction across sentence boundaries.

By treating simple conjunctions not just as vocabulary words, but as explicit signals of latent logical operations, this research direction provides a scalable, elegant pathway for teaching machines to reason about the connective tissue of human language.

Key Foundational Work

Learning Connective-based Word Representations for Implicit Discourse Relation Identification (2016) — contributed 2 concept elements to this direction.
Machine Comprehension with Discourse Relations (2015) — contributed 1 concept elements to this direction.
Closing the Gap: Domain Adaptation from Explicit to Implicit Discourse Relations (2015) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discourse_marker_prediction_59887103,
  title = {discourse marker prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-marker-prediction/rdr_598871035b754d1193ebb9cea6ae2c2e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_discourse_marker_prediction_59887103_v1,
  title = {discourse marker prediction},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-marker-prediction/rdr_598871035b754d1193ebb9cea6ae2c2e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake