Towards Domain-Independent Text Structuring Trainable on Large Discourse Treebanks

Grigorii Guz, Giuseppe CareniniEMNLP 2020 · DOI

Abstract

Text structuring is a vital component of Natural Language Generation (NLG), bridging content determination and surface realization. Traditional methods often rely on domain-specific rules or flat sequence ordering, failing to capture the hierarchical nature of discourse. This paper presents DepStructurer, a domain-independent model that simultaneously orders and organizes Elementary Discourse Units (EDUs) into a discourse tree using neural dependency induction and pointer networks. To better evaluate such models, we introduce Blocked Kendall’s $\tau$, a metric that prioritizes local semantic coherence. Experiments on the large-scale MEGA-DT treebank demonstrate that DepStructurer significantly outperforms sequence-based baselines, particularly on long documents.

Introduction

In the pipeline of Natural Language Generation, text structuring is the critical middle step: after deciding what to say (content determination) but before deciding how to say it (realization), a system must determine the order and logical flow of information. Historically, text structuring has been either hard-coded for specific domains or treated as a simple sequence-ordering problem. However, treating it as a flat sequence ignores the hierarchical nature of human discourse. Furthermore, standard evaluation metrics like Kendall’s $\tau$ are often too brittle, heavily penalizing a model if it moves a perfectly coherent paragraph to the wrong part of a document. This research argues that a robust structuring system should understand how small units of text (EDUs) cluster into larger, semantically cohesive structures.

Methodology: The DepStructurer Model

The authors frame text structuring as the simultaneous task of ordering $n$ unordered EDUs and inducing an unlabelled Rhetorical Structure Theory (RST) dependency tree. Their solution, DepStructurer, operates in two main stages:

  1. Initial Tree Induction: Using contextualized embeddings from a pre-trained ALBERT model, the system learns a "compatibility matrix" that scores potential edges between EDUs. It then uses the Chu-Liu-Edmonds algorithm to find the highest-weighting dependency tree.
  2. Ordering Children: Because a single node in a dependency tree can have multiple children on the same side, the basic tree structure doesn't fully determine the final linear order. To solve this, the model employs a Pointer Network specifically trained to order these groups of children.
The two-stage process of DepStructurer: dependency tree induction followed by child ordering.

Evaluation: Blocked Kendall’s $\tau$

A key contribution of this work is a new metric: Blocked Kendall’s $\tau$. Unlike the standard version, which counts every individual transposition, the "Blocked" version first merges correctly ordered sub-sequences into single units. This ensures that if a model correctly clusters a group of related ideas but places that group in the wrong section, it isn't penalized as harshly as a model that completely scrambles the ideas. This reflects the intuition that local coherence (keeping related ideas together) is often more important for readability than global positioning.

Illustration of the Blocked Kendall’s $\tau$ metric, which rewards the preservation of coherent semantic blocks.

Datasets and Experimental Setup

The study utilizes the MEGA-DT treebank, a massive "silver-standard" dataset containing approximately 250,000 discourse trees. These trees were generated via distant supervision from Yelp customer reviews. This dataset provides the scale necessary to train deep neural models for discourse, which was previously impossible with small, hand-annotated datasets like the RST-DT. By forcing the model to build a tree, the authors provide a "useful inductive bias," teaching the model that certain EDUs belong together in subtrees.

Key Results

  • Superior Ordering: On the full test set, DepStructurer achieved a Blocked $\tau$ of 52.7, significantly outperforming the standard Pointer Network baseline (40.4).
  • Long Document Performance: The advantage held even for long documents (over 35 EDUs), where DepStructurer scored 31.5 compared to the Pointer Network's 24.1.
  • Tree Quality: The model achieved an Unlabelled Attachment Score (UAS) of 25.8, showing it can induce meaningful discourse structures from scratch without being given the correct order as input.
  • Qualitative Success: Analysis showed that DepStructurer successfully grouped related EDUs into subtrees (e.g., keeping all comments about "food" together), whereas pure ordering models often scattered related points throughout the text.

Limitations and Future Work

While promising, the current model only predicts the unlabelled structure of the tree and does not yet account for relationship labels (e.g., Contrast vs. Elaboration) or "nuclearity" (identifying core vs. supporting messages). Future work will focus on integrating DepStructurer into full end-to-end NLG pipelines and verifying the correlation of Blocked Kendall’s $\tau$ with actual human judgments of text coherence.

Summary updated

BibTeX
@inproceedings{guz2020towards,
  author = {Grigorii Guz and Giuseppe Carenini},
  title = {{Towards Domain-Independent Text Structuring Trainable on Large Discourse Treebanks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  doi = {10.18653/v1/2020.findings-emnlp.281},
}
Figure