An End-to-End Document-Level Neural Discourse Parser Exploiting Multi-Granularity Representations

Ke Shi, Zhengyuan Liu, Nancy F. ChenarXiv 2020 · OpenReview

Abstract

Document-level discourse parsing, in accordance with the Rhetorical Structure Theory (RST), remains notoriously challenging. Challenges include the deep structure of document-level discourse trees, the requirement of subtle semantic judgments, and the lack of large-scale training corpora. To address such challenges, we propose to exploit robust representations derived from multiple levels of granularity across syntax and semantics, and in turn incorporate such representations in an end-to-end encoder-decoder neural architecture for more resourceful discourse processing. In particular, we first use a pre-trained contextual language model that embodies high-order and long-range dependency to enable finer-grain semantic, syntactic, and organizational representations. We further encode such representations with boundary and hierarchical information to obtain more refined modeling for document-level discourse processing. Experimental results show that our parser achieves the state-of-the-art performance, approaching human-level performance on the benchmarked RST dataset.

Introduction

Discourse parsing is the task of uncovering the hidden structural map of a text—identifying how small segments of text (Elementary Discourse Units, or EDUs) relate to one another to form a coherent message. While sentence-level parsing has seen great success, document-level parsing remains a significant challenge in NLP due to three main hurdles:

  1. Deep Structures: Documents have much deeper discourse trees than sentences (averaging 56 EDUs), making long-range dependencies harder to track.
  2. Error Propagation: In traditional top-down parsing, a single incorrect split at the top of the document can ruin every subsequent branch.
  3. Implicit Cues: Many relationships are not signaled by obvious connective words, requiring subtle semantic judgments.

Methods

The proposed architecture is built on three pillars: multi-granularity representations, boundary information, and a novel decoding strategy.

Multi-Granularity Representations

The model utilizes XLNet as a backbone. Unlike BERT, which is limited to 512 tokens, XLNet can handle up to 2048 tokens, making it suitable for full documents. The architecture encodes text at three levels: token level, EDU level, and global document level.

Architecture of the proposed end-to-end neural discourse parser.

Boundary Information

The researchers found that the "edges" of a text span are highly informative for discourse cues. They explicitly concatenate the embeddings of the first and last tokens of an EDU into its representation. Analysis showed that left boundaries are predictive of rhetorical relations (where words like "although" often appear), while right boundaries are better at predicting tree structure (likely due to punctuation).

Instead of common depth-first approaches, the model employs top-down breadth-first decoding. To mitigate error propagation, a layer-wise beam search is implemented. This allows the model to maintain several "best guesses" for an entire layer of the tree, choosing the most globally probable structure.

Results

The model was evaluated on the RST Discourse Treebank (RST-DT), consisting of 385 documents from the Wall Street Journal. Performance was measured across Span (S), Nuclearity (NS), and Relation (R) metrics.

Performance comparison of the proposed model against traditional and neural baselines.

The model achieved new state-of-the-art (SOTA) performance across all standard metrics. Notably, on Relation classification, the model reached an F1 score of 63.7, approaching the human agreement level of 65.4. The addition of layer-wise beam search provided a consistent 0.3–0.4 F1 boost, confirming that global optimization is beneficial for complex document structures.

Conclusion

This work demonstrates that combining high-capacity language models with explicit boundary information and global decoding strategies can significantly improve document-level discourse parsing. While the model approaches human performance on gold-segmented EDUs, real-world applications will require robust automatic segmentation. Future work includes applying these representations to cross-lingual parsing and downstream tasks like abstractive summarization.

Summary updated

Abstract

Document-level discourse parsing, in accordance with the Rhetorical Structure Theory (RST), remains notoriously challenging. Challenges include the deep structure of document-level discourse trees, the requirement of subtle semantic judgments, and the lack of large-scale training corpora. To address such challenges, we propose to exploit robust representations derived from multiple levels of granularity across syntax and semantics, and in turn incorporate such representations in an end-to-end encoder-decoder neural architecture for more resourceful discourse processing. In particular, we first use a pre-trained contextual language model that embodies high-order and long-range dependency to enable finer-grain semantic, syntactic, and organizational representations. We further encode such representations with boundary and hierarchical information to obtain more refined modeling for document-level discourse processing. Experimental results show that our parser achieves the state-of-the-art performance, approaching human-level performance on the benchmarked RST dataset.
BibTeX
@misc{shi2020end,
  author = {Ke Shi and Zhengyuan Liu and Nancy F. Chen},
  title = {{An End-to-End Document-Level Neural Discourse Parser Exploiting Multi-Granularity Representations}},
  year = {2020},
  eprint = {2012.11169},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2012.11169},
}
Figure