Integrating Discourse Context Beyond Single Sentences

27 concepts25 papers1 proposal

For decades, natural language processing treated the sentence as a self-contained atom of meaning. Models parsed syntax, classified sentiment, and translated text under the assumption that a single string of words contained all the information necessary for its own interpretation. This assumption creates a fundamental bottleneck known as contextual isolation.

In reality, human language is a continuous tapestry. The meaning, syntax, and rhetorical function of any given sentence are deeply contingent on the broader discourse. When models isolate sentences, they lose the threads that bind them together. As Werner Frey et al. observed in Automatic Construction of a Knowledge Base by Analysing Texts in Natural Language, a coherent information model cannot be constructed if entities and facts introduced in one sentence are referenced blindly in another.

This limitation was brought into sharp focus by Denis Paperno et al. through The LAMBADA dataset: Word prediction requiring a broad discourse context. The authors demonstrated that many language tasks become completely impossible when a model is restricted to local syntax, highlighting a critical need for architectures that prioritize broad narrative integration over isolated sentence processing.

Structural and Referential Blind Spots

The most immediate casualty of contextual isolation is structural parsing. In many languages, critical grammatical elements are routinely omitted if they can be inferred from the environment. For example, in pro-drop languages like Chinese, "zero pronouns" leave sentences without explicit subjects. As Qingyu Yin et al. demonstrate in A Deep Neural Network for Chinese Zero Pronoun Resolution, resolution models frequently fail at sentence boundaries because they lack access to the preceding history of potential antecedents. Similarly, Yufei Chen et al. show in Pre- and In-Parsing Models for Neural Empty Category Detection that detecting these unpronounced elements requires cross-sentence discourse context to resolve structural ambiguities.

Early parsing frameworks also suffered from this limitation. In Transforming Syntactic Graphs into Semantic Graphs, Hae-Chang Rim et al. argued that integrating context across multiple sentences is necessary to maintain a continuous narrative state, allowing systems to prune hundreds of invalid interpretations that appear plausible in isolation.

Beyond missing pronouns, tracking how entities move in and out of a listener's attention—known as salience—requires a continuous model of discourse. Eva Hajicová et al. explored this in Topic-focus and Salience, showing that a sentence must anchor itself to preceding information to maintain coherence. Without this anchoring, document-level tasks like parsing regulatory texts fail; as Nikhil Dinesh et al. note in Computing Logical Form on Regulatory Texts, legal requirements are riddled with cross-references and anaphoric links that span multiple paragraphs or sections.

The Pragmatic Shift: Meaning is Context-Dependent

Contextual isolation doesn't just break syntax; it distorts semantic and pragmatic interpretation. A sentence that looks objective in isolation might be highly subjective when placed in a broader argument. Janyce Wiebe et al. demonstrated in Development and Use of a Gold-Standard Data Set for Subjectivity Classifications that resolving ambiguous cases of subjectivity requires incorporating information from preceding utterances and document-level patterns.

Consider the challenge of identifying generic expressions (e.g., "Lions are predators") versus specific assertions. In Discourse-sensitive Automatic Identification of Generic Expressions, Annemarie Friedrich shows that models cannot effectively capture how a writer transitions between general rules and specific facts without considering the sequence of preceding statements. Similarly, the restrictiveness of a noun phrase modifier often depends on information provided earlier in a paragraph rather than the sentence itself, a phenomenon analyzed by Gabriel Stanovsky et al. in Annotating and Predicting Non-Restrictive Noun Phrase Modifications.

[Preceding Discourse Context] ---> Clarifies ---> [Target Sentence]
       |                                                 |
       +---> Resolves Subjectivity                       +---> Isolated: Ambiguous
       +---> Determines Generic vs. Specific             +---> Isolated: Misclassified
       +---> Identifies Rhetorical Fallacies             +---> Isolated: Unnoticed

This dependency intensifies in complex rhetorical tasks:

Architectural Paradigms for Discourse Modeling

To break out of sentence-level isolation, researchers have designed architectures that explicitly model the flow of discourse rather than relying on simple text concatenation.

Latent Segment and Sequence Modeling

Instead of classifying sentences independently, models must capture the overarching document structure. Qianli Ma et al. addressed this in A Span-based Dynamic Local Attention Model for Sequential Sentence Classification by using a supervised dynamic local attention mechanism to group contiguous sentences into coherent semantic blocks. This allows the model to learn how rhetorical functions shift across a document.

Multi-Hop Iterative Reasoning

In document-level machine translation, simple context windows often fail to capture nuanced transitions. To solve this, Long Zhang et al. introduced the Multi-Hop Transformer for Document-Level Machine Translation. This architecture mimics a human draft-editing process, iteratively reasoning over multiple previous sentences to resolve coreference and polysemy while avoiding exposure bias.

Discourse Grammars

Rather than treating logical links between sentences as isolated occurrences, Emily Pitler et al. demonstrate in Easily Identifiable Discourse Relations that adjacent discourse relations follow statistically significant patterns. Modeling these transitions as a structured grammar allows systems to resolve highly ambiguous or implicit structural links.

Impact on Downstream Applications

Shifting from sentence-level to discourse-level modeling has fundamentally altered how downstream NLP applications are framed:

Ultimately, to build systems capable of deep understanding, semantic annotation frameworks must also evolve. As Stephen Helmreich et al. argued in Interlingual Annotation of Multilingual Text Corpora, focusing annotation solely on individual sentences neglects cross-sentence phenomena like anaphora resolution and discourse relations, which are foundational for true language-independent semantic representation.

Key Foundational Work

Automatic Construction of a Knowledge Base by Analysing Texts in Natural Language (1983) — contributed 1 concept elements to this direction.
Automatic Construction of a Knowledge Base by Analysing Texts in Natural Language (1983) — contributed 1 concept elements to this direction.
Transforming Syntactic Graphs into Semantic Graphs (1990) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_discourse_level_context_dependency_04b5d02c,
  title = {discourse-level context dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-level-context-dependency/rdr_04b5d02c8a2449b6a31285673646892f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_discourse_level_context_dependency_04b5d02c_v1,
  title = {discourse-level context dependency},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/discourse-level-context-dependency/rdr_04b5d02c8a2449b6a31285673646892f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake