The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error

Katherine Atwell, Anthony Sicilia, Seong Jae Hwang, Malihe Alikhani

Abstract

Discourse parsing models, primarily trained on news corpora, frequently suffer from significant performance degradation when applied to out-of-distribution (OOD) domains such as Reddit or fiction. While measuring distribution shift is a standard practice, traditional metrics often fail to predict the actual increase in parser error. This paper introduces the $h$-discrepancy—a classifier-aware metric—as a superior predictor of the "error-gap." Through a large-scale empirical study involving over 2,400 experiments across diverse datasets, the authors demonstrate that the $h$-discrepancy provides a robust, conservative estimate of model failure, significantly outperforming standard statistical measures like Maximum Mean Discrepancy (MMD).

Introduction

Discourse parsing is the task of identifying the semantic and structural relationships between text segments, such as "Contrast," "Cause," or "Expansion." While modern models achieve respectable performance on benchmark datasets like the Wall Street Journal (RST-DT), their utility in real-world applications is often limited by domain shift. When a model trained on formal news text encounters the informal structure of a Reddit thread or the narrative flow of fiction, its accuracy typically plummets.

Historically, researchers have relied on general measures of distribution shift to anticipate these failures. However, these measures often lack a direct link to the specific logic of the classifier. This paper addresses this gap by proposing that the impact of a domain shift depends not just on the data itself, but on how that shift interacts with the model's hypothesis space.

The Limitations of Standard Distribution Metrics

The central problem in domain adaptation is the "error-gap": the difference between a model's performance on training data versus unseen test data. A common assumption is that a larger statistical distance between two datasets (distribution shift) will naturally lead to a larger error-gap.

However, the authors argue that standard "two-sample" statistics—such as Maximum Mean Discrepancy (MMD) or Energy distance—are poor proxies for performance. These metrics analyze raw data distributions in isolation from the task. Consequently, they can signal a massive shift that has no impact on the classifier's decision boundaries, or conversely, miss a subtle shift that fundamentally breaks the model’s logic.

The $h$-discrepancy: A Classifier-Aware Approach

To provide a more reliable estimate of the error-gap, the authors introduce the $h$-discrepancy (or source-guided discrepancy). Unlike MMD, the $h$-discrepancy is "classifier-aware." It functions by searching the model's hypothesis space to find a classifier that agrees with the current model on the source data but disagrees maximally on the target data.

If such a hypothesis exists, it indicates that the domain shift is "meaningful" to the model, suggesting that the current predictions on the target domain are unreliable. The authors provide a theoretical foundation for this approach (Theorem 1), proving that the $h$-discrepancy acts as a conservative bound on the bias introduced by domain shift.

Experimental Framework

The study's empirical breadth is significant, encompassing over 2,400 experiments. The authors utilized six major discourse datasets to cover a spectrum of genres:

  • News: RST-DT, PDTB 2.0, and PDTB 3.0.
  • Biomedical: BioDRB.
  • Spoken Word: TED-MDB (transcribed TED talks).
  • Multi-genre: The GUM corpus (including Reddit, interviews, and how-to guides).

To ensure consistency across different frameworks, the authors mapped labels between the Rhetorical Structure Theory (RST) and Penn Discourse Treebank (PDTB) schemas. They also employed Ordinary Least Squares (OLS) regression to isolate the effects of specific variables, such as embedding types and genre, on the resulting error-gap.

Key Findings

1. Superior Predictive Power

The $h$-discrepancy proved to be a far more accurate predictor of parser failure than traditional metrics. It achieved a Spearman rank correlation of 0.8299 with the actual error-gap, whereas MMD and Energy distance trailed significantly at 0.5051 and 0.6059, respectively. In "within-distribution" settings, the $h$-discrepancy’s linear correlation reached an impressive 0.9519.

2. The Impact of Genre and Representation

The research highlighted that not all domains are equally difficult to adapt to. Regression analysis showed that Fiction and How-to Guides present the greatest challenges for transfer learning, while Academic Journals and Biographies are relatively easier.

Furthermore, the choice of text representation matters:

  • S-BERT (Sentence-BERT): More effective for domain adaptation when the source and target domains are relatively similar.
  • A-BERT (Average-BERT): More robust and reliable when the domains are highly divergent.

3. Model Complexity

The authors found that more complex models (e.g., multi-layer fully-connected networks) generally handle domain shift better than simpler linear classifiers. This suggests that increased modeling capacity allows the parser to learn more generalized features that survive the transition between domains.

Conclusion

This research demonstrates that the $h$-discrepancy is a powerful tool for researchers and practitioners who need to deploy discourse parsers in the real world. By moving beyond simple data-distance metrics and adopting a classifier-aware approach, it becomes possible to accurately estimate when a model is likely to fail. While limitations remain—such as the noise introduced by cross-framework label mapping—the $h$-discrepancy sets a new standard for measuring the impact of domain shift in NLP. The authors have released their code and a Python package to facilitate the adoption of these metrics in the wider research community.

Summary updated

Abstract

Discourse analysis allows us to attain inferences of a text document that extend beyond the sentence-level. The current performance of discourse models is very low on texts outside of the training distribution's coverage, diminishing the practical utility of existing models. There is need for a measure that can inform us to what extent our model generalizes from the training to the test sample when these samples may be drawn from distinct distributions. While this can be estimated via distribution shift, we argue that this does not directly correlate with change in the observed error of a classifier (i.e. error-gap). Thus, we propose to use a statistic from the theoretical domain adaptation literature which can be directly tied to error-gap. We study the bias of this statistic as an estimator of error-gap both theoretically and through a large-scale empirical study of over 2400 experiments on 6 discourse datasets from domains including, but not limited to: news, biomedical texts, TED talks, Reddit posts, and fiction. Our results not only motivate our proposal and help us to understand its limitations, but also provide insight on the properties of discourse models and datasets which improve performance in domain adaptation. For instance, we find that non-news datasets are slightly easier to transfer to than news datasets when the training and test sets are very different. Our code and an associated Python package are available to allow practitioners to make more informed model and dataset choices.
BibTeX
@inproceedings{atwell2022change,
  author = {Katherine Atwell and Anthony Sicilia and Seong Jae Hwang and Malihe Alikhani},
  title = {{The Change that Matters in Discourse Parsing: Estimating the Impact of Domain Shift on Parser Error}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  doi = {10.18653/v1/2022.findings-acl.68},
}
Figure