Discourse Probing of Pretrained Language Models

Fajri Koto, Jey Han Lau, Timothy Baldwin

Abstract

While Large Language Models (LLMs) have demonstrated remarkable sentence-level fluency, their ability to comprehend the "big picture" of document-level discourse remains an open question. This paper systematically investigates how seven popular pretrained models—spanning encoder-only, decoder-only, and encoder-decoder architectures—capture discourse information across four languages. Using a probing framework, the researchers find that models like BART and RoBERTa are particularly adept at representing discourse structure, though this knowledge is often concentrated in specific layers. The study highlights that training objectives and architectural choices are more critical for discourse understanding than sheer data volume.

Beyond Syntax: The Challenge of Discourse

Most research into "probing"—the practice of testing what a model knows internally—has focused on syntax, such as part-of-speech tagging or sentence structure. However, human language is not merely a sequence of independent sentences; it is a cohesive web of ideas known as discourse.

While models like BERT excel at predicting the next word, do they truly understand how a paragraph is organized? Can they recognize the rhetorical relationship between two clauses, such as identifying if one serves as an "explanation" for the other? This paper addresses the lack of systematic research into how pretrained language models (PLMs) capture document-level discourse across different architectures and languages.

Probing Methodology and Discourse Tasks

To investigate these questions, the authors employed a technique called Probing. Instead of fine-tuning the entire model for a specific task, they froze the weights of the pretrained models (including BERT, GPT-2, and BART) and extracted the embeddings from each of the 12 layers. A simple Multi-Layer Perceptron (MLP) classifier was then trained on these frozen representations. High performance by this classifier indicates that the model had already "learned" discourse information during its original pretraining phase.

Overview of discourse tasks including EDU segmentation, nuclearity, and relation prediction.

The study evaluated models across English, Mandarin Chinese, German, and Spanish using several benchmarks:

  • RST-DT & CDTB: Treebanks based on Rhetorical Structure Theory (RST) used for predicting nuclearity (identifying the "core" vs. "supplementary" parts of a sentence) and specific relations (e.g., cause, contrast).
  • XSUM & Wikipedia: Used for Next Sentence Prediction and Sentence Ordering tasks.
  • DisSent: A dataset for predicting discourse connectives like because or although.
  • Cloze Story Test: A task requiring models to select the correct ending to a four-sentence story.

Comparative Analysis of Model Architectures

The results revealed that architecture and training objectives play a more significant role in discourse understanding than the size of the training dataset.

Average normalized scores across layers for various language models, showing how discourse knowledge is distributed.

Key Insights:

  • BART as the Discourse Leader: BART’s encoder was the top performer overall. Interestingly, its decoder performed significantly worse, suggesting that the objective of generating text might distract a model from understanding underlying structure.
  • Architecture vs. Data: Despite being trained on 750GB of data, T5 performed worse than RoBERTa and BART, which were trained on 160GB. This suggests that the specific pretraining objective is the primary driver of discourse proficiency.
  • The ALBERT Spike: In most models, discourse knowledge fluctuates across layers. However, in ALBERT, discourse information is nearly non-existent until the very last layer, where it suddenly spikes.
  • Cross-Lingual Consistency: The "shape" of how models capture discourse is remarkably consistent across languages. Universal structural cues appear to be picked up similarly whether the model is processing German, Chinese, or English.

Layer-wise Dynamics and Task Complexity

The probing revealed distinct patterns in where discourse knowledge resides within a model. For BERT, this information is best captured in the middle-to-late layers (7–10), while for ELECTRA, the peak occurs at the very end (layers 10–12). RoBERTa consistently outperformed the original BERT, validating that improved pretraining recipes—such as removing the Next Sentence Prediction objective—actually enhance structural understanding.

However, a gap remains between "easy" and "hard" tasks. While models are excellent at Next Sentence Prediction (accuracies > 0.80), they struggle with Sentence Ordering (Spearman’s $\rho$ ~ 0.40). This indicates that while models can identify local coherence between two sentences, they struggle to grasp the logical flow of an entire paragraph.

Conclusion

This research demonstrates that while current PLMs possess significant discourse knowledge, it is highly dependent on architecture and localized within specific layers. Future work is needed to explore larger models and address the relative weakness of decoders (like GPT-2) in capturing structural relationships. Additionally, tasks like EDU segmentation remain challenging, suggesting that models have yet to perfect the representation of idea boundaries.

Summary updated

Abstract

Existing work on probing of pretrained language models (LMs) has predominantly focused on sentence-level syntactic tasks. In this paper, we introduce document-level discourse probing to evaluate the ability of pretrained LMs to capture document-level relations. We experiment with 7 pretrained LMs, 4 languages, and 7 discourse probing tasks, and find BART to be overall the best model at capturing discourse -- but only in its encoder, with BERT performing surprisingly well as the baseline model. Across the different models, there are substantial differences in which layers best capture discourse information, and large disparities between models.
BibTeX
@inproceedings{koto2021discourse,
  author = {Fajri Koto and Jey Han Lau and Timothy Baldwin},
  title = {{Discourse Probing of Pretrained Language Models}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2021},
  url = {https://openreview.net/forum?id=iQvB12RKzt7},
}
Figure