TL;DR
This paper introduces a novel multi-token mask-filling approach for identifying implicit discourse relations between sentences. By enabling language models to predict multi-word connectives instead of single words, the method significantly improves accuracy and reduces ambiguity, offering a more nuanced understanding of how sentences connect.
What Problem Does This Paper Address?
Understanding how sentences relate to each other (discourse relations) is crucial for natural language understanding. While explicit connectives like "but" or "however" signal these relationships, many sentences rely on implicit connections. Previous work used masked language models (MLMs) like BERT to predict single-token connectives in these implicit cases. However, this approach suffered from ambiguity, as single words can have multiple meanings. For instance, "but" can signal various comparison or contrast relations. This paper tackles the limitation of single-token fillers by proposing and evaluating a method that predicts multi-token connectives, aiming for greater precision and less ambiguity in identifying discourse relations.
Methods
The core of this work is the adaptation of the Extended-Matrix (EMAT) decoder approach, originally developed for question-answering, to predict multi-token discourse connectives. The EMAT decoder extends an MLM's output vocabulary by adding embeddings for multi-token phrases. This allows the model to predict sequences of tokens as fillers for masked positions without retraining the entire pre-trained MLM. The authors specifically adapted this to predict discourse connectives from a curated list. They then used this multi-token mask-filling model for two main experiments: first, to compare its performance against single-token mask-filling on general implicit discourse relations, and second, to investigate the role of preposed syntactic structures in signaling these relations.
Datasets
The primary dataset used is the Penn Discourse Treebank (PDTB-3), a widely used corpus for discourse relation annotation. For training their multi-token completion model, they utilized a large subset of the Wikipedia English dataset (~838K sentence pairs). For the second experiment, they also incorporated data from DiscoGeM 1.0, a corpus containing political speeches, literature, and Wikipedia texts, which is annotated at the sentence level.
Key Insights and Ideas
- Multi-token connectives are less ambiguous: The paper provides strong empirical evidence that multi-token connectives (e.g., "in contrast," "for example") are inherently less ambiguous than single-token ones, mapping to fewer distinct discourse senses. This is a key motivation for the proposed approach.
- Preposing aids discourse relation signaling: The study confirms and extends previous findings that preposed syntactic structures (like phrases at the beginning of a sentence) help language models identify discourse relations. This effect is even more pronounced when using less ambiguous multi-token connectives.
- EMAT decoder is efficient for multi-token prediction: The adaptation of the EMAT decoder allows for the prediction of multi-token phrases without the need to retrain the entire base MLM, making the approach computationally efficient.
Key Results
- Improved Accuracy: The multi-token mask-filling model significantly outperforms the single-token baseline in recognizing implicit discourse relations, achieving higher accuracy across different sense levels (e.g., a@1 of 87.72% for multi-token vs. 70.93% for single-token at Level 3).
- Reduced Ambiguity: The model's predictions are associated with substantially fewer discourse senses compared to the single-token baseline, indicating a reduction in ambiguity. For instance, in the preposed setting, the average sense count for correct predictions was 4.93 for their model versus 13.79 for the single-token baseline.
- Preposing Benefits: The multi-token model shows higher accuracy and confidence on preposed structures compared to their canonical counterparts, reinforcing the hypothesis that preposing aids in signaling discourse coherence.
Limitations and Future Work
- Limited Annotated Data: The current work relies on explicit discourse relations to construct training and test sets for implicit relations. The authors suggest that crowd-sourcing implicit discourse relation annotations could provide more useful resources.
- Test Dataset Size: The test dataset, while substantial (~16K samples), could be expanded. The authors also note the potential of other corpora like the Georgetown University Multilayer Corpus (GUM), though its RST-style annotation might require conversion.
- Predefined Connective Set: The model is trained on a predefined set of connectives. The authors acknowledge that the frequency of connectives in the training data might not perfectly reflect their natural occurrence, potentially leading to under- or overrepresentation.
- Single Sense Assumption: The experiments primarily considered a single gold sense per relation. Future work could explore recognizing multiple senses simultaneously, especially since the PDTB-3 allows for annotating two connectives and senses.
- Exploring Other Syntactic Structures: The authors suggest extending their work to assess the discourse relevance of other non-canonical syntactic structures, such as post-posing and right extraposition.
Summary updated