Abstract
Zeyu Dai, Ruihong Huang. Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers).
2018.
Introduction
Most NLP models attempt to understand the logical relationship between two sentences by analyzing them in isolation. However, this approach ignores the broader context of the paragraph. This paper argues that discourse relations are not independent; rather, they form a "flow" across a paragraph. By modeling entire paragraphs and their inter-dependencies using a hierarchical Bi-LSTM and a Conditional Random Field (CRF) layer, the authors achieve a new state-of-the-art in identifying implicit discourse relations.
In discourse analysis, we categorize how Discourse Units (DUs) are logically connected—such as Comparison, Contingency, Expansion, or Temporal. While machines excel at identifying "explicit" relations marked by connectives like because or but, "implicit" relations (where the connective is missing) remain a significant challenge. Previous methods treated sentence pairs as independent islands, failing to capture how the meaning of a sentence is shaped by its neighbors or how discourse patterns recur across a text.
Methods
The authors propose a paragraph-level neural network designed to capture these inter-dependencies through three primary innovations:
1. Hierarchical Paragraph Encoding
Instead of encoding isolated sentence pairs, the model processes the entire paragraph. A word-level Bi-LSTM creates representations for each word, which are then max-pooled into a single vector for each DU. Subsequently, a DU-level Bi-LSTM processes the sequence of these units, allowing each sentence's representation to be influenced by its preceding and following context.
2. Untied Parameters
The model recognizes that identifying explicit relations (often a word disambiguation task) requires a different strategy than identifying implicit ones (which requires deep semantic understanding). The authors "untie" the final softmax layer, training separate classifiers for explicit and implicit relations within the same shared network.
3. CRF Layer for Global Consistency
To model the logical "flow" of a paragraph, a Conditional Random Field (CRF) layer is added on top of the prediction layer. This allows the model to consider the sequence of relations as a whole, learning that certain relations are likely to follow others (e.g., a sequence of Temporal relations), ensuring global consistency across the paragraph's discourse structure.
Key Results
The model was evaluated on the Penn Discourse Treebank (PDTB) v2.0, focusing on the four top-level senses: Comparison, Contingency, Expansion, and Temporal. The proposed system significantly outperformed previous state-of-the-art models.
Key performance highlights from the evaluation include:
- State-of-the-Art Performance: The full model (including the CRF layer) achieved a Macro F1-score of 48.82% for implicit relations, a significant jump over previous models that hovered around 40-46%.
- Ablation Study: As shown in the results table, the addition of "Untie Parameters" and the "CRF Layer" provided incremental but significant gains in both Macro F1 and Accuracy.
- Minority Class Boost: The model showed dramatic improvements in difficult categories. In binary classification tests, it improved the F1-score for Comparison by 6 points and Temporal by 7 points compared to previous systems.
- Ensemble Performance: By using an ensemble of 10 models, the Macro F1-score for implicit relations reached 51.84%.
Discussion and Limitations
The study demonstrates that context is paramount; a sentence's pragmatic meaning is defined by its role within the paragraph. Furthermore, identifying explicit relations serves as a useful anchor for predicting implicit ones in the same context. However, some limitations remain:
- Paragraph Length: The model's advantage is most pronounced in longer paragraphs. In short paragraphs (only 2 DUs), where there is no wider context to exploit, the performance gain over simpler models is reduced.
- CRF Constraints: The CRF layer is optimized for sequences and is less suitable for the "one-versus-all" binary classification tasks common in older literature.
- Granularity: While successful at the top-level senses, future work is needed to determine if paragraph-level modeling can assist with fine-grained sub-senses (e.g., distinguishing between "Cause" and "Pragmatic Cause").
Summary updated