Abstract
We argue that semantic meanings of a sentence or clause can not be interpreted independently from the rest of a paragraph, or independently from all discourse relations and the overall paragraph-level discourse structure. With the goal of improving implicit discourse relation classification, we introduce a paragraph-level neural networks that model inter-dependencies between discourse units as well as discourse relation continuity and patterns, and predict a sequence of discourse relations in a paragraph. Experimental results show that our model outperforms the previous state-of-the-art systems on the benchmark corpus of PDTB.
Introduction
Discourse relation classification is a fundamental task in natural language processing that involves identifying the logical connections between segments of text, known as Discourse Units (DUs). While explicit relations—signaled by connective words like "but" or "because"—can be identified with high accuracy (often exceeding 90%), implicit relations remain a significant challenge. These relations lack explicit markers, requiring models to infer meaning from context alone.
Traditional approaches often treat discourse relations as isolated pairs, ignoring the broader paragraph context. This "tunnel vision" fails to capture how the meaning of a sentence is shaped by its neighbors and the overall flow of the discourse. This paper proposes a paragraph-level approach to capture these inter-dependencies.
Methods
The proposed architecture is a hierarchical neural network designed to process paragraphs as coherent sequences rather than isolated pairs.
Hierarchical Bi-LSTM
The model employs a two-tier Bi-LSTM structure to "read" a paragraph much like a human does:
- Word-Level Bi-LSTM: Processes individual words to generate representations for each DU.
- DU-Level Bi-LSTM: Takes the sequence of DU representations as input. This allows the model to capture the "flow" of information across the paragraph, where the representation of a specific DU is influenced by both preceding and succeeding units.
Untied Prediction Parameters and CRF Layer
To further refine the model, the authors introduced two key components:
- Untied Parameters: Recognizing that explicit and implicit relations have different characteristics, the model uses separate classifiers for each. This prevents the easier explicit relations from dominating the learning process.
- CRF Layer (Conditional Random Field): A CRF layer is added on top of the prediction layer to model the global continuity of discourse relations. It captures statistical patterns in how relations follow one another (e.g., a "Comparison" relation might frequently follow an "Expansion").
Experimental Setup
The model was evaluated on the Penn Discourse Treebank (PDTB) v2.0, focusing on the four top-level semantic classes: Comparison (Comp), Contingency (Cont), Expansion (Exp), and Temporal (Temp). The dataset includes 2,159 Wall Street Journal articles, and the authors reconstructed paragraph contexts to facilitate sequence-based modeling.
Key Results
The paragraph-level model demonstrated substantial improvements over previous state-of-the-art systems and pair-wise baselines.
As shown in the results table, the full model (including untied parameters and the CRF layer) achieved a Macro F1 score of 48.82% for implicit relations. This represents a significant gain over earlier models like Lan et al. (2017) at 47.80%. Notably, the model showed strong performance in difficult categories such as "Comparison" and "Temporal," where F1 scores improved by approximately 6 to 7 points over baseline systems. When using an ensemble of ten models, the performance climbed even higher to a Macro F1 of 51.84%.
Discussion and Conclusion
The success of this approach highlights the importance of context in discourse analysis. By modeling the entire paragraph, the system treats explicit relations as "landmarks" that help clarify the pragmatic roles of surrounding sentences. The DU representations become "context-aware," reflecting the dynamic nature of meaning within a narrative or argument. Future work could explore integrating higher-level document structures or specialized discourse-aware embeddings to further enhance the model's understanding of complex textual relationships.
Reference: Dai, Z., & Huang, R. (2018). Improving Implicit Discourse Relation Classification by Modeling Inter-dependencies of Discourse Units in a Paragraph. arXiv:1804.05918v1 [cs.CL].
Summary updated