Abstract: While pretrained language models (LMs) excel at sentence-level syntax, their ability to grasp the "big picture" of document structure remains less understood. This study probes seven popular models across four languages to evaluate their discourse-level knowledge. Findings reveal that BART (specifically its encoder) and RoBERTa are the most proficient at capturing discourse, while pure decoders like GPT-2 struggle. Furthermore, discourse information is primarily localized in the deeper layers of these models.
Introduction
Most research into the internal knowledge of language models focuses on local syntax and semantics, such as subject-verb agreement or entity identification. However, human communication relies on discourse—the complex way sentences link together to form a coherent narrative or argument. This paper addresses a significant gap: understanding how models capture document-level relations, such as whether one sentence provides "evidence" for another or if a sequence of events is in the correct order.
Methodology
The researchers employed a probing methodology. Instead of fine-tuning entire models, they kept the layers "frozen" and trained a simple Multi-Layer Perceptron (MLP) classifier on top of the hidden representations from each individual layer. This approach reveals what information is naturally present at different depths.
Using the Rhetorical Structure Theory (RST) framework (illustrated above), the authors tested seven models (BERT, RoBERTa, ALBERT, ELECTRA, GPT-2, BART, and T5) across seven distinct discourse tasks:
- 4-way NSP: Selecting the correct next sentence from four options.
- Sentence Ordering: Reconstructing the order of a shuffled paragraph.
- Discourse Connective Prediction: Identifying missing conjunctions (e.g., because, however).
- RST Nuclearity: Identifying the "nucleus" (main point) versus the "satellite" (supporting info).
- RST Relation: Identifying specific relationships like elaboration or cause.
- RST EDU Segmentation: Breaking sentences into Elementary Discourse Units.
- Cloze Story Test: Choosing the most logical ending to a short story.
Key Results and Discussion
The probing results across all tasks and layers provide several critical insights into the architecture of discourse knowledge in LMs.
The Encoder Advantage
There is a stark divide between model architectures. Encoders (like BERT and RoBERTa) and the encoder-half of BART are significantly more effective at discourse tasks than decoders. This suggests that the training objective of generating text (causal language modeling) does not necessarily force a model to understand high-level document structure as effectively as bidirectional encoding does.
Discourse is "Deep"
For complex tasks like the Cloze Story Test and RST Relation prediction, performance generally improves as one moves into the deeper layers of the model. High-level story understanding and logical flow are not found in the early layers that handle basic word meanings; rather, they emerge as the model processes more context.
The ALBERT Paradox
ALBERT, which utilizes parameter sharing, shows a unique pattern: it performs poorly in its early and middle layers but exhibits a massive spike in discourse knowledge in its final layer. This suggests that ALBERT may be deferring complex structural integration until the very end of its processing pipeline.
Cross-Lingual Consistency
Strikingly, the "shape" of discourse knowledge across layers is nearly identical across English, Chinese, German, and Spanish. This suggests that the way Transformer architectures learn to organize document-level information is a fundamental property of the architecture and pretraining process, independent of the specific language.
Conclusion
BART's encoder and RoBERTa emerged as the overall champions of discourse, likely due to their robust pretraining objectives. However, even the best models struggle with Sentence Ordering, indicating that truly understanding the logical flow of long paragraphs remains a significant challenge. Future work may involve designing new pretraining objectives specifically aimed at these structural weaknesses to produce models capable of more coherent long-form generation.
Summary updated