Learning Discourse Structure from Scarce Data

156 concepts135 papers2 proposals

When we read a text, we do not just process a sequence of independent sentences. We automatically construct a mental map of how those sentences relate to one another—identifying that one statement causes another, a second turn in a conversation concedes a point, or a specific clause provides background information.

In computational linguistics, mapping this underlying architecture is known as discourse parsing. Whether using frameworks like Rhetorical Structure Theory (RST) or the Penn Discourse Treebank (PDTB), understanding these relationships is vital for downstream tasks like document summarization, sentiment analysis, and dialogue modeling.

However, building models to parse discourse faces a severe roadblock: expert annotation scarcity. Unlike token-level tasks (such as part-of-speech tagging) or sentence-level tasks (such as sentiment classification), discourse annotation requires analyzing long-range, hierarchical dependencies across entire documents or messy, multi-party conversations. As noted by Shafiq Joty et al. and Amir Zeldes et al., this process demands highly specialized linguistic expertise, making the creation of gold-standard datasets exceptionally expensive, time-consuming, and difficult to scale.

This creates a stark data bottleneck. While modern deep learning architectures require massive datasets to generalize effectively, expert-annotated discourse corpora remain small and fragmented, as highlighted by Gregor Weiss. To bridge this gap, researchers have developed a variety of creative strategies to bypass the manual annotation bottleneck, training robust models by extracting structural signals from unlabeled text, alternative tasks, and generative frameworks.


The Challenge of the Unspoken

The data scarcity problem is most acute when discourse relationships are implicit. When a text contains explicit connectives (e.g., "I stayed inside because it was raining"), a model can rely on surface keywords. But when those connectives are missing (e.g., "I stayed inside. It was raining."), the model must infer the logical link from semantics alone.

Because human-labeled examples of these implicit relations are rare, early approaches sought to "hallucinate" the missing data. Zhi-Min Zhou et al. demonstrated that language models could be used to predict the most likely missing connective between two clauses, effectively turning an implicit classification problem into an explicit one with minimal training data.

A similar bottleneck occurs when trying to resolve missing information within sentences, such as omitted pronouns (zero pronouns) or unstated event participants (implicit semantic roles). Manually mapping these hidden references across long document contexts is incredibly labor-intensive (Jennifer Sikos et al.; Longyue Wang et al.). To overcome this, Carina Silberer reimagined implicit role linking as an anaphora resolution task, creating a heuristic method to automatically "manufacture" training data from standard, widely available corpora.


Bootstrapping via Weak and Distant Supervision

If human labels are too expensive to produce, can we find alternative signals that act as a proxy for discourse structure?

One approach is weak supervision or data programming. For instance, Lila Gravellier et al. used a weakly supervised framework to adapt discourse segmenters from polished written text to messy, spoken multi-party conversations, achieving significant performance gains without requiring massive, manually labeled conversational datasets.

Another powerful strategy is distant supervision, which leverages existing labels from a related task to infer discourse properties. Consider document-level sentiment analysis. Parminder Bhatia et al. demonstrated that a document's overall sentiment is fundamentally tied to its rhetorical structure; "important" parts of a text (nuclei) carry more sentiment weight than purely contextual parts (satellites).

Patrick Huber et al. inverted this intuition to solve the data scarcity problem. Instead of using discourse trees to predict sentiment, they used massive collections of user reviews (which naturally come with star ratings) and back-propagated the sentiment signals to automatically infer the underlying hierarchical discourse trees. This distant supervision approach yielded MEGA-DT, a "silver-standard" discourse treebank of 250,000 documents—orders of magnitude larger than any existing human-annotated corpus.

[Distant Supervision Intuition]
Standard Flow:  Discourse Tree Structure  ───► Weights Text Importance ───► Predicts Sentiment
Huber et al.:   Massive Labeled Sentiment ───► Backpropagates Signal   ───► Infers Discourse Tree

Generative and Structural Reformulations

Traditional discourse parsers treat the problem as a sequence of rigid classification decisions: predicting whether a boundary exists, and then selecting a relation label from a fixed index. When training data is scarce, these classifiers struggle to generalize.

Recent work bypasses this by reformulating discourse parsing as a generative task. The authors of this generative framework developed a Language Model-based Generative Classifier (LMGC) that linearizes hierarchical discourse trees into plain text strings. By embedding official label definitions directly into the text, the model can leverage the vast, pre-trained semantic knowledge of a language model to "write" the parse tree, rather than learning abstract classification boundaries from scratch.

Taking this generative intuition a step further, Wei Li transformed discourse dependency parsing into a code generation task. By representing complex text relationships as Python-like code, the author enabled Large Language Models (LLMs) to use their highly optimized code-generation capabilities to parse discourse structures, bypassing the need for specialized, data-hungry discourse architectures.

This generative shift also applies to dialogue. Deyi Xiong et al. tackled ellipsis and co-reference resolution in task-oriented dialogues by training a model to generate pragmatically complete sentences from incomplete user utterances, converting a sparse structural classification problem into a text-to-text translation task.


Self-Supervised and Multi-Task Representation Learning

When explicit labels are entirely absent, models can learn the implicit "flow" of discourse through self-supervised pretext tasks.

For example, Bhargavi Paranjape et al. trained a hierarchical encoder to predict the bag-of-words of surrounding utterances in large, unlabeled conversational corpora. By forcing the model to guess what context comes next, it naturally learns deep conversational representations that can later be fine-tuned for dialogue act or emotion classification with very few human labels. Similarly, Hirokazu Kiyomaru et al. used contrastive self-supervised learning to strip away surface-level lexical details, forcing sentence encoders to focus purely on a sentence's functional and relational role within a broader narrative.

[Contrastive Self-Supervised Learning]
Anchor Sentence:      "The rain started pouring."
Positive Context:     "So we opened our umbrellas."  (High similarity in embedding space)
Negative Context:     "Quantum physics is complex."  (Pushed far away in embedding space)
Result: Model learns the functional, relational link between sentences without human labels.

When labels are available but sparse, contrastive learning can maximize their utility. Yuxin Jiang et al. introduced the GOLF framework for implicit relation recognition. GOLF uses contrastive learning to align text representations not just with a single label, but with the global hierarchy of all possible labels, squeezing more structural awareness out of limited training examples.

Additionally, multi-task learning allows models to pool resources across fragmented datasets. Jonathan May demonstrated that training a model on multiple distinct discourse labeling schemas simultaneously allows it to leverage cross-schema correlations. This shared statistical strength significantly improves the model's ability to identify rare, underrepresented rhetorical functions that standard data augmentation techniques miss.


Cross-Lingual and Cross-Corpus Projection

Finally, researchers handle data scarcity in specific domains or languages by borrowing structure from where it is abundant.

In multilingual settings, annotation projection allows researchers to transfer discourse knowledge across parallel corpora. Elena Chistova demonstrated that end-to-end top-down parsers can transfer effectively from resource-rich languages to target languages with minimal native data. This cross-lingual alignment was also leveraged by Marian Marchal et al. to build DiscoNaija, a discourse-annotated parallel corpus for Nigerian Pidgin, mapping structural coherence rules from English to a low-resource language.

Even within the same language, discourse datasets are often fragmented by different annotation styles. To combat this, Yi Cheng et al. converted three distinct Chinese discourse corpora into a unified Dependency Discourse Structure (DDS) framework. By merging these disparate resources into a single representation, they proved that aggregating data across different annotation philosophies significantly boosts parsing performance for small-scale, data-starved domains.

History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 135 papers
  2. Version 1 · 2026-10-04 · 109 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_discourse_data_scarcity_53e7fb00,
  title = {discourse data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-data-scarcity/rdr_53e7fb00d5dc481e894896bdb9d14e79},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_discourse_data_scarcity_53e7fb00_v2,
  title = {discourse data scarcity},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-data-scarcity/rdr_53e7fb00d5dc481e894896bdb9d14e79/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake