TL;DR
This paper tackles the challenge of identifying "alternative lexicalizations" (discourse signals not in the standard connective list) in text, which are crucial for understanding discourse structure but suffer from limited labeled data. They propose a novel weakly-supervised learning approach that leverages unlabeled data to significantly improve performance, achieving state-of-the-art results.
What Problem Does This Paper Address?
Understanding how sentences and paragraphs connect is fundamental to natural language understanding. Discourse parsing aims to identify these connections, often signaled by "discourse connectives" (like "because," "and," "if-then"). However, language is flexible, and people also use "alternative lexicalizations" (AltLex) – phrases like "this means," "for that reason," or "it all adds up to" – to signal these same relationships. While explicit connectives are well-studied, AltLex are less common and harder to identify, leading to a scarcity of labeled data for training models. This paper addresses this data scarcity by developing a weakly-supervised learning method to improve AltLex identification.
Methods
The authors propose a two-phase, cyclic self-supervised learning approach:
- Phase 1 (Recall-focused): This phase prioritizes finding as many potential AltLex signals as possible. They use a sequence labeling model (based on RoBERTa) fine-tuned on the Penn Discourse Treebank (PDTB). To boost recall, they adjust the loss function to down-weight the "None" class (which represents no signal). This phase extracts "fuzzy" signal candidates.
- Candidate Disambiguation: A separate model is trained to discriminate between true AltLex signals and phrases that are not discourse signals. This model helps filter out incorrect predictions from the first phase.
- Leveraging Unlabeled Data: Crucially, they integrate unlabeled news text (from the New York Times corpus) into the training. They extract confident predictions from the model on this unlabeled data and use them as additional training samples. This process is iterated, allowing the model to learn from its own successful predictions.
- Phase 2 (F1-Score focused): After the cyclic learning, a final model is trained on the combined labeled and pseudo-labeled data, this time optimizing for the F1 score (balancing precision and recall).
A key architectural choice is adapting the models to work at the paragraph level, rather than just sentence level, to better capture the context that AltLex signals often rely on. They use a three-class encoding: single signals, multi-word signals, and no signal.
Datasets
- Penn Discourse Treebank (PDTB) v3: Used for initial labeled training and evaluation. This dataset contains both explicit connectives and alternative lexicalizations.
- New York Times Annotated Corpus (NYT): A large corpus of news articles used as unlabeled data to augment training. A subset of documents from 2000-2002 was used.
Key Insights and Ideas
- Weak supervision is powerful for AltLex: The core idea that leveraging unlabeled data can significantly boost performance for a data-scarce task like AltLex identification is validated.
- Paragraph-level context is crucial: Moving beyond sentence-level analysis is essential for accurately identifying AltLex, as these phrases often link ideas across sentence boundaries.
- A two-phase approach balances recall and precision: The strategy of first optimizing for recall (finding many candidates) and then using a disambiguation model to filter them is effective.
- AltLex and explicit connectives are blurry: The authors observe that their model sometimes confuses AltLex with explicit connectives, suggesting that future work might benefit from models that handle both types jointly.
Key Results
- State-of-the-art performance: The proposed weakly-supervised approach achieves state-of-the-art results on AltLex identification.
- Significant improvement over baseline: Their novel architecture, when trained with unlabeled data, improves performance by about 5% (both exactly and partially matched) compared to a baseline model trained only on labeled data.
- Ensemble benefits: Combining the outputs of three randomly initialized models further improved performance and reduced variance.
- Impact of unlabeled data size: Performance generally increased with the amount of unlabeled data used, up to 10,000 documents.
Limitations and Future Work
- Paragraph structure: The model struggles with texts lacking clear paragraph structure, suggesting a need for pre-processing or a more robust sentence-based model.
- Continuous signals only: The current model only identifies continuous AltLex phrases, excluding discontinuous ones (e.g., "the more... the more"). This might impact performance on genres like speeches or debates.
- Blurring lines between AltLex and explicit connectives: The authors suggest future work should explore models that combine the identification of both explicit connectives and AltLex, acknowledging the fuzzy boundaries between them.
- Upper performance bound: The exact upper performance limit of their approach was not identified due to computational constraints.
Summary updated