Abstract
In implicit discourse relation classification, the goal is to predict the logical relationship between adjacent sentences in the absence of overt discourse connectives. This task is notoriously difficult for both humans and machines, leading to a shortage of high-quality annotated data. This paper presents a novel distant supervision pipeline that leverages the latent knowledge within large pre-trained language models to predict missing connectives, effectively transforming implicit relations into explicit ones. The resulting model achieves state-of-the-art performance in unsupervised settings and demonstrates remarkable cross-domain robustness.
Introduction
Discourse relations—the logical links that hold a text together—are essential for natural language understanding. While some relations are "explicit," signaled by connectives such as because, but, or however, many are "implicit." For instance, in the sequence "It's raining. I'm staying home," a causal link is understood despite the lack of a formal connective.
Identifying these implicit relations is a major challenge in Natural Language Processing (NLP). The subjectivity of manual annotation has led to a data bottleneck, where supervised models struggle due to a lack of labeled training examples. This research addresses this bottleneck by introducing a method that requires no manually labeled implicit data, relying instead on "distant supervision" through connective prediction.
Methodology: The Explicitation Pipeline
The authors propose a two-step pipeline that simplifies the classification of implicit relations by leveraging the abundance of explicit discourse data.
1. Explicitation via Language Modeling
Given a pair of sentences, the model first generates several "explicit" versions by inserting various one-word connectives (e.g., and, since, but). A large pre-trained language model (LM), such as RoBERTa-large or GPT-2 XL, is then used to score these candidates. The LM determines which connective fits most naturally within the specific context. This step effectively uses the LM as a bridge, translating the implicit problem into an explicit format.
2. Explicit Relation Classification
The highest-scoring connective (or a weighted average of all candidates) is then passed to a classifier. Crucially, this classifier is trained exclusively on explicit discourse relations, which are plentiful and easy to extract from large corpora. Because the classifier is highly accurate at identifying relations when a connective is present, the success of the system depends on the LM's ability to provide a meaningful "explicitation."
Data and Evaluation
The model was evaluated using two primary datasets:
- PDTB 2.0 (Penn Discourse Treebank): The standard benchmark for discourse analysis. The model was trained on explicit relations and tested on implicit relations (Sections 21-22).
- BioDRB (Biomedical Discourse Relation Bank): A specialized dataset used for cross-domain evaluation. This tested whether a model trained on news text (PDTB) could generalize to the complex logic of biomedical research articles.
Key Insights
- Latent Discourse Knowledge: A significant finding is that modern LMs already possess a deep understanding of sentence-level logic from their general pre-training. They can often predict the correct missing connective without any task-specific fine-tuning.
- The Power of Marginalization: Instead of relying on a single "best" connective, the authors found that averaging (marginalizing) over the entire distribution of possible connectives improved robustness, especially in cases where the top-ranked choice was ambiguous.
- Comparison Relations: The model showed exceptional improvement in "Comparison" relations (e.g., contrast). This suggests that LMs are particularly sensitive to the linguistic cues that signal shifts in perspective or contradictions.
Results and Performance
The proposed method established a new state-of-the-art for unsupervised discourse classification:
- Benchmark Success: The model achieved a top 4-way classification F-score of 41.10 using GPT2-large, outperforming previous unsupervised benchmarks.
- Contrastive Logic Gains: For "Comparison" relations, the F-score nearly doubled from ~25% in previous work to 49.52%, significantly narrowing the gap with fully supervised models (58.35%).
- Cross-Domain Robustness: The model maintained strong performance on the BioDRB dataset without additional tuning. It even outperformed some supervised models in zero-shot scenarios, proving that the method captures fundamental linguistic logic rather than just domain-specific patterns.
Limitations and Future Directions
While successful, the approach is currently limited to 65 one-word connectives, potentially missing the nuance provided by multi-word markers like "on the other hand." Furthermore, the system's performance is inherently capped by the quality of the underlying language model.
Future research will explore expanding the connective list and using this explicitation task as a diagnostic tool to evaluate how well new language models understand context. Additionally, the authors suggest this pipeline could be used to generate high-quality synthetic data to bolster supervised learning in the future.
Summary updated