Evaluating Discourse in Structured Text Representations

Elisa Ferracane, Greg Durrett, Junyi Jessy Li, Katrin Erk

Abstract

Discourse structure is integral to understanding a text and is helpful in many NLP tasks. Learning latent representations of discourse is an attractive alternative to acquiring expensive labeled discourse data. Liu and Lapata (2018) propose a structured attention mechanism for text classification that derives a tree over a text, akin to an RST discourse tree. We examine this model in detail, and evaluate on additional discourse-relevant tasks and datasets, in order to assess whether the structured attention improves performance on the end task and whether it captures a text's discourse structure. We find the learned latent trees have little to no structure and instead focus on lexical cues; even after obtaining more structured trees with proposed model modifications, the trees are still far from capturing discourse structure when compared to discourse dependency trees from an existing discourse parser. Finally, ablation studies show the structured attention provides little benefit, sometimes even hurting performance.

Introduction

Understanding the discourse structure of a document—how sentences logically and rhetorically connect—is highly beneficial for NLP tasks like summarization and sentiment analysis. However, manually annotating discourse data is incredibly expensive and requires trained linguists. To bypass this, recent research has proposed using structured attention mechanisms to automatically induce a latent dependency tree over a document while training on standard downstream tasks like text classification. This paper evaluates whether these induced latent trees actually capture real discourse structure or are merely mathematical artifacts.

Methodology

The study focuses on the Liu and Lapata (L&L) model, which employs a bidirectional LSTM and structured attention (via the Matrix-Tree theorem) to induce non-projective dependency trees over sentences.

The L&L model architecture featuring document-level structured attention.

To test the model's robustness, several modifications were introduced:

  1. Logic Corrections: A masking bug in the original code was fixed, and information flow was adjusted so semantic updates percolate from children to parents.
  2. Architectural Ablations: The document-level biLSTM was removed in some experiments to force the model to rely on the tree structure. Multiple levels of tree percolation were also added.
  3. Ground Truth Comparison: Induced trees were compared against parsed Rhetorical Structure Theory (RST) trees.

Datasets

The models were evaluated on five datasets, including three new tasks designed to test discourse sensitivity:

  • Yelp: 5-way sentiment classification.
  • Debates: Binary classification of congressional floor debates.
  • Writing Quality (WQ): Classification of NYT science articles.
  • Writing Quality with Topic Control (WQTC): WQ controlled for topic to prevent lexical "cheating."
  • Wall Street Journal Sentence Order (WSJSO): Discriminating between correctly ordered and jumbled documents.

Analysis of Latent Structures

A primary finding is the "Vacuous Tree" phenomenon. Instead of learning complex discourse hierarchies, the model typically selects the first or last sentence as the root and attaches every other sentence directly to it.

A visualization of a "vacuous tree" where the model fails to learn hierarchical discourse structure.

These flat structures suggest the model uses the root node to hunt for strong lexical cues (e.g., "stars" in Yelp or "oppose" in Debates) rather than understanding document flow. While removing the biLSTM forced deeper trees, it often reduced downstream accuracy, indicating the biLSTM was previously acting as a performance crutch.

Key Results

Quantitative analysis confirms the structural deficiency of induced trees:

  • Tree Height: Learned trees were shallow (height 2.0–4.0) compared to real RST trees (height ~25.0).
  • Leaf Density: 82% to 95% of nodes in learned trees were leaves, whereas real RST trees have only about 56% leaf nodes.
  • Task Performance: Structured attention rarely improved accuracy and sometimes degraded it. On the WSJSO task, 100% of generated trees were vacuous.

Conclusion

While architectural modifications can force deeper tree induction, the resulting structures still fail to align with linguistic reality. Purely unsupervised latent tree induction via downstream classification appears insufficient for capturing true discourse. Future work may require direct supervision or ground-truth constraints to guide neural networks toward meaningful document structuring.

Summary updated

Abstract

Discourse structure is integral to understanding a text and is helpful in many NLP tasks. Learning latent representations of discourse is an attractive alternative to acquiring expensive labeled discourse data. Liu and Lapata (2018) propose a structured attention mechanism for text classification that derives a tree over a text, akin to an RST discourse tree. We examine this model in detail, and evaluate on additional discourse-relevant tasks and datasets, in order to assess whether the structured attention improves performance on the end task and whether it captures a text's discourse structure. We find the learned latent trees have little to no structure and instead focus on lexical cues; even after obtaining more structured trees with proposed model modifications, the trees are still far from capturing discourse structure when compared to discourse dependency trees from an existing discourse parser. Finally, ablation studies show the structured attention provides little benefit, sometimes even hurting performance.
BibTeX
@inproceedings{ferracane2019evaluating,
  author = {Elisa Ferracane and Greg Durrett and Junyi Jessy Li and Katrin Erk},
  title = {{Evaluating Discourse in Structured Text Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=a73TBwrP1gj},
}
Figure