TL;DR
This paper tackles the challenging task of identifying implicit discourse relations (like "because" or "however" but without explicit markers) by augmenting neural networks with external knowledge from ConceptNet and coreference resolution. They found that this knowledge significantly improves accuracy, especially when combined with a broader contextual understanding of the text.
What Problem Does This Paper Address?
Discourse relations are crucial for understanding the logical connections between sentences and paragraphs, enabling tasks like summarization and question answering. While explicit discourse relations (marked by words like "because," "but," "therefore") are relatively easy to detect, implicit discourse relations, which lack these explicit cues, pose a significant challenge. Identifying these implicit links requires understanding world knowledge and subtle contextual clues that are difficult for standard neural networks to learn from limited annotated data. The paper aims to bridge this gap by explicitly providing external knowledge to neural models.
Methods
The core of the proposed method is a Knowledge-Augmented Gated Recurrent Unit (MAGE-GRU). This is an extension of the standard Gated Recurrent Unit (GRU) that can incorporate external linguistic knowledge. Instead of just processing a sequence of words, MAGE-GRU takes a directed acyclic graph as input. This graph augments the word sequence with edges representing relationships like antonymy or coreference, derived from external knowledge sources.
Key aspects of their approach include:
- MAGE-GRU: This model uses a special vector
gtthat selectively combines past hidden states based on detected external knowledge signals (e.g., antonyms between words). This allows the model to create "shortcuts" and better capture long-range dependencies. - External Knowledge Sources:
- ConceptNet: A large, open-source knowledge graph containing common sense relations between words and concepts. They specifically leverage relations like "Antonym" and "Causes" which are relevant to discourse relations.
- Coreference Resolution: Identifying mentions that refer to the same entity (e.g., pronouns and their antecedents). This helps connect arguments to relevant contextual information.
- Input Formatting: They combine the two arguments (Arg1 and Arg2) into a single input sequence and also include a text chunk that precedes the arguments in the paragraph. Special tags (
<Arg1>,</Arg1>, etc.) are used to delineate these parts.
Datasets
The primary dataset used is the Penn Discourse TreeBank (PDTB). This is a widely used corpus for discourse relation annotation. The authors use the standard 11-way classification of implicit discourse relations. They evaluate their model using two common splits:
- Cross Validation: A 10-fold cross-validation setup using sections 0-24 of the PDTB.
- Most-used Split: A standard train/development/test split (sections 2-21 for training, 22 for development, 23 for testing).
Key Insights and Ideas
- External Knowledge is Crucial for Implicit Relations: The paper strongly suggests that world knowledge, particularly semantic relationships like antonymy, is essential for understanding implicit discourse relations. Standard word embeddings alone, even when trained on large corpora, struggle to distinguish antonyms due to their similar distributional properties.
- Context Matters, But Differently for Explicit vs. Implicit: While including the preceding text chunk (context) significantly improved performance for explicit discourse relations, it offered no significant gain for implicit ones. This suggests that implicit relations rely more on semantic knowledge and less on broader textual context compared to explicit relations.
- MAGE-GRU's Effectiveness: The MAGE-GRU architecture, by explicitly incorporating external knowledge as graph edges, proves effective in creating shortcuts and improving the model's ability to capture relevant semantic links.
- Limitations of Standard Word Embeddings for Antonymy: The paper highlights that word embeddings alone are insufficient for recognizing antonymy, a key signal for many discourse relations.
Key Results
- Significant Accuracy Gains: The proposed MAGE-GRU model, augmented with ConceptNet and coreference resolution, achieved state-of-the-art or competitive results on the PDTB. For instance, the "Paragraph+ConceptNet" configuration achieved an accuracy of 0.4083 (±0.011) in Cross Validation, outperforming previous LSTM-based models.
- ConceptNet's Impact: Augmenting the model with ConceptNet consistently improved performance, with the gain being statistically significant.
- Coreference Resolution's Limited Impact: Coreference resolution alone did not provide a significant boost for implicit discourse relation classification in their experiments.
- Contextual Information: The "Paragraph" input format (including preceding text) improved performance when combined with external knowledge, but the gain was not always statistically significant for implicit relations.
Limitations and Future Work
- Noise in Knowledge Bases: The authors acknowledge that external knowledge bases like ConceptNet can contain noise or questionable triplets, which can sometimes lead to misclassifications. They mitigated this by filtering triplets based on weights and using nearest neighbors for word embeddings.
- Low-Frequency Relations: The model tended to ignore very low-frequency relations, a common issue in NLP. They suggest leveraging unlabeled corpora to address this.
- Event Coreference: The paper points out that standard coreference resolution (nominal and pronominal) might not be sufficient; resolving event coreference could be beneficial.
- Future Directions:
- Bidirectional MAGE-GRU: Extending the MAGE-GRU architecture to be bidirectional.
- Other Knowledge Sources: Exploring alternative knowledge bases (e.g., Freebase) and high-level NLP analyses (e.g., event coreference).
- Combining with Other Models: Leveraging the composability of MAGE-GRU to integrate it with other neural network architectures.
Summary updated