Abstract
Discourse parsing models, primarily trained on news corpora, frequently suffer from significant performance degradation when applied to out-of-distribution (OOD) domains such as Reddit or fiction. While measuring distribution shift is a standard practice, traditional metrics often fail to predict the actual increase in parser error. This paper introduces the $h$-discrepancy—a classifier-aware metric—as a superior predictor of the "error-gap." Through a large-scale empirical study involving over 2,400 experiments across diverse datasets, the authors demonstrate that the $h$-discrepancy provides a robust, conservative estimate of model failure, significantly outperforming standard statistical measures like Maximum Mean Discrepancy (MMD).
Introduction
Discourse parsing is the task of identifying the semantic and structural relationships between text segments, such as "Contrast," "Cause," or "Expansion." While modern models achieve respectable performance on benchmark datasets like the Wall Street Journal (RST-DT), their utility in real-world applications is often limited by domain shift. When a model trained on formal news text encounters the informal structure of a Reddit thread or the narrative flow of fiction, its accuracy typically plummets.
Historically, researchers have relied on general measures of distribution shift to anticipate these failures. However, these measures often lack a direct link to the specific logic of the classifier. This paper addresses this gap by proposing that the impact of a domain shift depends not just on the data itself, but on how that shift interacts with the model's hypothesis space.
The Limitations of Standard Distribution Metrics
The central problem in domain adaptation is the "error-gap": the difference between a model's performance on training data versus unseen test data. A common assumption is that a larger statistical distance between two datasets (distribution shift) will naturally lead to a larger error-gap.
However, the authors argue that standard "two-sample" statistics—such as Maximum Mean Discrepancy (MMD) or Energy distance—are poor proxies for performance. These metrics analyze raw data distributions in isolation from the task. Consequently, they can signal a massive shift that has no impact on the classifier's decision boundaries, or conversely, miss a subtle shift that fundamentally breaks the model’s logic.
The $h$-discrepancy: A Classifier-Aware Approach
To provide a more reliable estimate of the error-gap, the authors introduce the $h$-discrepancy (or source-guided discrepancy). Unlike MMD, the $h$-discrepancy is "classifier-aware." It functions by searching the model's hypothesis space to find a classifier that agrees with the current model on the source data but disagrees maximally on the target data.
If such a hypothesis exists, it indicates that the domain shift is "meaningful" to the model, suggesting that the current predictions on the target domain are unreliable. The authors provide a theoretical foundation for this approach (Theorem 1), proving that the $h$-discrepancy acts as a conservative bound on the bias introduced by domain shift.
Experimental Framework
The study's empirical breadth is significant, encompassing over 2,400 experiments. The authors utilized six major discourse datasets to cover a spectrum of genres:
- News: RST-DT, PDTB 2.0, and PDTB 3.0.
- Biomedical: BioDRB.
- Spoken Word: TED-MDB (transcribed TED talks).
- Multi-genre: The GUM corpus (including Reddit, interviews, and how-to guides).
To ensure consistency across different frameworks, the authors mapped labels between the Rhetorical Structure Theory (RST) and Penn Discourse Treebank (PDTB) schemas. They also employed Ordinary Least Squares (OLS) regression to isolate the effects of specific variables, such as embedding types and genre, on the resulting error-gap.
Key Findings
1. Superior Predictive Power
The $h$-discrepancy proved to be a far more accurate predictor of parser failure than traditional metrics. It achieved a Spearman rank correlation of 0.8299 with the actual error-gap, whereas MMD and Energy distance trailed significantly at 0.5051 and 0.6059, respectively. In "within-distribution" settings, the $h$-discrepancy’s linear correlation reached an impressive 0.9519.
2. The Impact of Genre and Representation
The research highlighted that not all domains are equally difficult to adapt to. Regression analysis showed that Fiction and How-to Guides present the greatest challenges for transfer learning, while Academic Journals and Biographies are relatively easier.
Furthermore, the choice of text representation matters:
- S-BERT (Sentence-BERT): More effective for domain adaptation when the source and target domains are relatively similar.
- A-BERT (Average-BERT): More robust and reliable when the domains are highly divergent.
3. Model Complexity
The authors found that more complex models (e.g., multi-layer fully-connected networks) generally handle domain shift better than simpler linear classifiers. This suggests that increased modeling capacity allows the parser to learn more generalized features that survive the transition between domains.
Conclusion
This research demonstrates that the $h$-discrepancy is a powerful tool for researchers and practitioners who need to deploy discourse parsers in the real world. By moving beyond simple data-distance metrics and adopting a classifier-aware approach, it becomes possible to accurately estimate when a model is likely to fail. While limitations remain—such as the noise introduced by cross-framework label mapping—the $h$-discrepancy sets a new standard for measuring the impact of domain shift in NLP. The authors have released their code and a Python package to facilitate the adoption of these metrics in the wider research community.
Summary updated