Towards Domain-Independent Supervised Discourse Parsing Through Gradient Boosting

Patrick Huber, Giuseppe CareniniarXiv 2022 · arXiv

TL;DR

This paper proposes a novel supervised discourse parsing framework that uses gradient boosting to overcome the chronic problem of domain dependency. By training a sequence of "weak" shift-reduce parsers, the authors aim to separate general, domain-independent discourse features from the specific nuances of small training datasets, resulting in a more robust and portable parser.

What Problem Does This Paper Address?

Discourse parsing—the task of uncovering the structural and semantic relationships between parts of a text—is vital for high-level NLP tasks like summarization and sentiment analysis. However, it faces a "severe data sparsity issue." The most popular treebanks, such as RST-DT and GUM, contain fewer than 400 documents each.

Because training data is so limited, modern deep-learning models tend to overfit to the specific domain of the training set (e.g., news articles). When these models are applied to a new domain (e.g., creative writing or technical manuals), their performance drops significantly. The authors tackle this "domain adaptation" challenge by rethinking how models learn from limited data.

Methods

The authors extend the current state-of-the-art (SOTA) neural shift-reduce parser in three key ways:

  1. Neural Gradient Boosting: Instead of training one large, complex model, they use an ensemble of "weak" classifiers. As shown in the schematic below, the process is iterative. The first weak classifier (with few parameters) is forced to learn simple, general discourse structures. Subsequent models are trained to correct the errors (the "loss") of the previous steps, capturing increasingly nuanced and potentially domain-specific features.

    Figure 1: Schematic of our proposed gradient boosted discourse parsing model. Grey=Inputs/Outputs, Green=Gradient Boosting component, Red/Orange/Yellow=Trained models. A12 refers to the second shift-reduce action of the first step. Please note that the models are independent and only connected by the aggregation in green.
  2. Linguistically Inspired Stack Representations: In shift-reduce parsing, the "stack" holds representations of text spans. Previous models truncated long spans by simply keeping the first and last tokens. This paper proposes a more sophisticated method: reducing stack elements based on sub-tree nuclearity. This ensures that the most important parts of a discourse unit (the "nuclei") are preserved, which is where discourse relations are linguistically theorized to hold.

  3. Full Tree Prediction: While some previous models focused only on the structure and importance (nuclearity) of text spans, this approach adds a dedicated component to predict the specific discourse relations (e.g., Contrast, Elaboration, Cause), resulting in complete discourse trees.

Datasets

The paper utilizes the two primary benchmarks in the field:

  • RST-DT (Rhetorical Structure Theory Discourse Treebank): A collection of Wall Street Journal articles, widely considered the standard but highly domain-specific.
  • GUM (Georgetown University Multilayer Corpus): A more diverse dataset containing multiple genres (interviews, news, travel guides, etc.), used here to test domain independence.

Key Insights and Ideas

  • The "Weak Classifier" as a Regularizer: The core hypothesis is that a model with very few parameters cannot overfit to complex, domain-specific noise. By starting with weak classifiers, the system is forced to learn the most frequent, "general" features of discourse first.
  • Separation of Concerns: By using a staged approach, the authors suggest there may be a "threshold" of boosting steps. Models before this threshold are likely domain-independent, while steps after it capture the specific quirks of the training dataset.
  • Nuclearity Matters: Moving away from arbitrary text truncation toward linguistically motivated stack reduction acknowledges that discourse structure is not just about word position, but about the hierarchy of information importance.

Key Results

As this paper presents a new paradigm and a "towards" approach, the primary contribution is the methodology and the planned evaluation framework. The authors propose evaluating their model across four dimensions:

  • Performance: Comparing the ensemble of weak classifiers against a single, strong SOTA model.
  • Efficiency: Investigating if the linear combination of smaller models allows for faster training on restricted hardware.
  • Parameter Sensitivity: Finding the optimal balance between the size of each weak classifier and the number of boosting steps.
  • Domain Independence: Quantifying how performance holds up across different domains as more boosting steps are added.

Limitations and Future Work

The authors note that their nuclearity-based stack reduction, while promising, still relies on heuristic assumptions that require deeper linguistic validation. The main future work involves executing the proposed four-dimensional evaluation to prove that gradient boosting effectively separates general discourse features from domain-specific ones. They hope this will eventually lead to a truly "universal" discourse parser that works across any text type without requiring massive new datasets.

Summary updated

Abstract

Discourse analysis and discourse parsing have shown great impact on many important problems in the field of Natural Language Processing (NLP). Given the direct impact of discourse annotations on model performance and interpretability, robustly extracting discourse structures from arbitrary documents is a key task to further improve computational models in NLP. To this end, we present a new, supervised paradigm directly tackling the domain adaptation issue in discourse parsing. Specifically, we introduce the first fully supervised discourse parser designed to alleviate the domain dependency through a staged model of weak classifiers by introducing the gradient boosting framework.
BibTeX
@misc{huber2022towards,
  author = {Patrick Huber and Giuseppe Carenini},
  title = {{Towards Domain-Independent Supervised Discourse Parsing Through Gradient Boosting}},
  year = {2022},
  eprint = {2210.09565},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2210.09565},
}
Figure