ToNy: Contextual embeddings for accurate multilingual discourse segmentation of full documents

Philippe Muller, Chloé Braud, Mathieu MoreyNAACL 2019

TL;DR

This paper introduces ToNy, a system for automatically segmenting full documents into elementary discourse units (EDUs), a crucial step for understanding text structure. By leveraging contextual embeddings like BERT, ToNy achieves state-of-the-art performance across multiple languages and discourse frameworks, even without relying on pre-defined sentence boundaries.

What Problem Does This Paper Address?

Discourse segmentation is the foundational task of identifying the minimal text units (EDUs) that form the building blocks of a document's structure. These EDUs are then linked by discourse relations to create a deeper understanding of how ideas connect. While previous work has achieved high accuracy on English newswire texts, these systems often rely on gold-standard sentence boundaries and are limited to specific frameworks like Rhetorical Structure Theory (RST). This paper addresses the challenge of creating a more generic and robust discourse segmenter that can handle full documents, work across different languages and discourse formalisms (RST, SDRT, PDTB), and crucially, operate without the assumption of perfect sentence segmentation. The authors highlight that sentence boundary detection itself is imperfect and can negatively impact discourse segmentation.

Methods

The core of ToNy is a sequence prediction architecture that utilizes pre-trained contextual embeddings. Specifically, they experiment with both ELMo and BERT, ultimately favoring BERT for its strong performance. The system is designed to handle full documents, meaning it doesn't require pre-segmentation into sentences. They compare two configurations: one that uses gold sentence boundaries as input and another that processes the entire document without any sentence information. The models are trained as sequence labelers, predicting EDU boundaries for each word. They also implement baseline systems, including rule-based segmenters and a bi-directional LSTM with randomly initialized or pre-trained word embeddings (FastText, GloVe).

Datasets

The paper leverages 15 corpora from the DisRPT shared task, covering 9 different languages and 3 discourse frameworks (RST, SDRT, PDTB). Notably, the Chinese PDTB corpus was not included due to unavailability. The datasets vary significantly in size, number of documents, and vocabulary, presenting a challenge for generalization. The authors provide statistics on these corpora, highlighting the differences in sentence and boundary counts, which inform the difficulty of the segmentation task for different datasets.

Key Insights and Ideas

  • Contextual Embeddings are Key: Pre-trained contextual embeddings, particularly BERT, significantly boost discourse segmentation performance, outperforming traditional word embeddings and even ELMo in many scenarios.
  • Sentence Boundaries are Not Always Necessary (and can be detrimental): The system demonstrates that it can achieve competitive or even superior results without relying on gold sentence boundaries, suggesting that robust models can learn to infer these boundaries implicitly. This is a significant finding as perfect sentence segmentation is not guaranteed.
  • Multilingual Models Offer Generality: The use of a multilingual BERT model allows for a more generic approach, enabling comparisons across languages and frameworks with a single model architecture.
  • Cross-Corpus Generalization is Challenging: While the models perform well on individual corpora, there's a notable drop in performance when transferring models between different RST datasets (e.g., GUM vs. RST-DT), indicating that annotation guidelines, even within the same framework, can vary significantly and impact generalization.

Key Results

ToNy, powered by BERT, achieves state-of-the-art F1 scores across most of the 14 evaluated datasets, with scores ranging from 81% to 96%. In the document-level setting (without gold sentence boundaries), BERT-based models consistently outperform baseline systems. For instance, on the English RST-DT dataset, they report an F1 score of 91.2% in the document-level setting, comparable to previous work. The paper also shows that while rule-based systems are simple and achieve decent results, they are generally outperformed by the neural models.

Limitations and Future Work

The authors acknowledge that discourse segmentation models exhibit only moderate generalization capabilities, even within the same language and discourse representation scheme. This is evidenced by the performance drop when testing on different RST corpora. They also note that the 512-token limit of BERT can be restrictive for very long, unsegmented documents, requiring preprocessing steps like sentence splitting. Future work could explore more expressive models, especially for connective identification, and conduct more controlled comparisons of language-specific versus multilingual BERT embeddings. The variability in annotation guidelines across corpora also presents an ongoing challenge for reproducible research in discourse analysis.

Summary updated

BibTeX
@inproceedings{muller2019tony,
  author = {Philippe Muller and Chloé Braud and Mathieu Morey},
  title = {{ToNy: Contextual embeddings for accurate multilingual discourse segmentation of full documents}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure