Evaluating Discourse in Structured Text Representations

Elisa Ferracane, Greg Durrett, Junyi Jessy Li, Katrin Erk

Abstract

Discourse structure is integral to understanding a text and is helpful in many NLP tasks. Learning latent representations of discourse is an attractive alternative to acquiring expensive labeled discourse data. Liu and Lapata (2018) propose a structured attention mechanism for text classification that derives a tree over a text, akin to an RST discourse tree. We examine this model in detail, and evaluate on additional discourse-relevant tasks and datasets, in order to assess whether the structured attention improves performance on the end task and whether it captures a text’s discourse structure. We find the learned latent trees have little to no structure and instead focus on lexical cues; even after obtaining more structured trees with proposed model modifications, the trees are still far from capturing discourse structure when compared to discourse dependency trees from an existing discourse parser. Finally, ablation studies show the structured attention provides little benefit, sometimes even hurting performance.

Introduction

In Natural Language Processing (NLP), understanding document organization—discourse structure—is vital for complex tasks like summarization and sentiment analysis. However, the high cost of manual discourse labeling has led researchers toward latent structure induction. These models aim to "discover" optimal tree structures for a document while training on downstream classification tasks. A prominent example is the structured attention model proposed by Liu and Lapata (2018). This study investigates whether such models truly learn linguistic discourse or merely identify mathematical shortcuts.

The Structured Attention Model

The evaluated model utilizes a structured attention mechanism based on the Matrix-Tree Theorem to compute marginal probabilities of dependency arcs between sentences. This process effectively induces a tree structure over the document.

Hierarchical architecture of the structured attention model at both word and sentence levels.

To improve upon the original design, the authors introduced a "percolation" modification. While the original model's information flow was ambiguous, the modified version explicitly forces information to flow from "children" nodes up to "parent" nodes, aligning more closely with linguistic composition.

Analysis of Latent Structures

A primary finding of this research is the "Vacuous Tree" phenomenon. Instead of complex hierarchical structures, the model frequently produces flat "star" shapes where a single root sentence connects to every other sentence in the document.

A "vacuous" tree where the root (sentence 1) is connected to all other sentences, failing to capture hierarchical discourse.

These trees lack the depth expected in human-annotated discourse. For instance, while gold-standard Rhetorical Structure Theory (RST) trees typically reach heights of ~25 for standard documents, the induced trees averaged heights of only 2 to 4. Furthermore, the model tends to select "roots" based on task-specific lexical cues—such as a star rating in a Yelp review or a stance-taking phrase in a Congressional debate—rather than rhetorical flow.

Performance and Ablation Studies

The study evaluated the model across five datasets: Yelp (sentiment), Debates (stance), Writing Quality (WQ), Writing Quality controlled for topic (WQTC), and sentence ordering (WSJSO).

Contrary to expectations, document-level structured attention did not consistently improve performance. In the Debates task, it actually hindered results. Ablation studies revealed that when a document-level BiLSTM is present, the structured attention provides negligible benefit. The BiLSTM appears to capture sufficient context, rendering the computationally expensive tree-induction redundant.

Conclusion

The findings suggest that purely unsupervised induction is insufficient for capturing true discourse structure. The models tend to be "lazy," opting for lexical shortcuts rather than complex linguistic organization if the task can be solved simply. Future research should explore injecting small amounts of ground-truth discourse data or applying tighter linguistic constraints to guide models toward more human-like structural representations.

Summary updated

Abstract

Discourse structure is integral to understanding a text and is helpful in many NLP tasks. Learning latent representations of discourse is an attractive alternative to acquiring expensive labeled discourse data. Liu and Lapata (2018) propose a structured attention mechanism for text classification that derives a tree over a text, akin to an RST discourse tree. We examine this model in detail, and evaluate on additional discourse-relevant tasks and datasets, in order to assess whether the structured attention improves performance on the end task and whether it captures a text’s discourse structure. We find the learned latent trees have little to no structure and instead focus on lexical cues; even after obtaining more structured trees with proposed model modifications, the trees are still far from capturing discourse structure when compared to discourse dependency trees from an existing discourse parser. Finally, ablation studies show the structured attention provides little benefit, sometimes even hurting performance.
BibTeX
@inproceedings{ferracane2019evaluating,
  author = {Elisa Ferracane and Greg Durrett and Junyi Jessy Li and Katrin Erk},
  title = {{Evaluating Discourse in Structured Text Representations}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2019},
  url = {https://openreview.net/forum?id=a73TBwrP1gj},
}

Loading related content...

Figure