Abstract
Zeyu Dai, Ruihong Huang. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP).
- Discourse parsing—identifying how sentences and clauses relate to one another—is notoriously difficult for AI because it often requires "reading between the lines" using commonsense knowledge. This paper introduces a knowledge regularization approach that integrates external event knowledge and entity coreference into neural networks. By adding a specialized loss term inspired by knowledge graph embeddings, the model learns to align its internal representations with real-world logic, significantly outperforming previous state-of-the-art systems on the PDTB benchmark.
Introduction
In natural language processing, discourse parsing is the task of identifying rhetorical relations (like Cause, Contrast, or Temporal) between text spans. While "explicit" relations are easy to spot thanks to connectives like "because" or "but," implicit relations are a significant challenge for machines.
Consider these two sentences:
- Tobacco consumption is rising.
- No smoking should be established as the norm.
A human easily identifies a "Cause" relation here because we know smoking causes lung cancer. However, a standard neural network might only see the word "lung cancer" a handful of times in its entire training set. This data sparsity makes it nearly impossible for models to learn these deep, commonsense connections from scratch. Previous attempts to fix this by "blindly" feeding in external facts often backfired, as the model would get distracted by irrelevant information that didn't fit the specific context.
Methodology
The authors build upon a hierarchical BiLSTM (Bidirectional Long Short-Term Memory) base model and introduce two major innovations:
- The Knowledge Layer: They insert a new layer between the word-level and discourse-unit-level BiLSTMs. This layer specifically modifies the hidden representations of words that participate in an "event" (e.g., smoking $\rightarrow$ causes $\rightarrow$ cancer) or a coreference chain (e.g., Steve and he referring to the same person).
- Knowledge Regularization: Instead of just treating external knowledge as a static feature, they incorporate it into the training objective. They adapt the TransE embedding method—which represents relations as translations in vector space ($head + relation \approx tail$)—into a regularization term.
This "tight integration" is the core contribution. The regularization term penalizes the model if its internal word representations don't align with known external facts, but it does so flexibly. This allows the model to "balance its attention" between the immediate text context and the external constraints.
Datasets and Knowledge Sources
The model was evaluated on the PDTB v2.0 (Penn Discourse Treebank), the gold standard for discourse parsing, containing ~19k explicit and ~17k implicit relations. External knowledge was integrated from several sources:
- ConceptNet: Used to extract causal and sub-event knowledge.
- Yao et al. (2018): A specialized dataset for temporal event knowledge.
- Stanford CoreNLP: Used to generate entity coreference clusters within the text.
Key Results
The proposed model achieved a new state-of-the-art on the PDTB dataset. Using ELMo embeddings, the full model reached a Macro F1 of 52.89% for implicit relations, a significant jump over the previous best of 51.06%.
Ablation studies confirmed that removing the regularization term caused performance to crater (dropping from ~50% to ~44% F1 in some tests), proving that the way knowledge is integrated is just as important as the knowledge itself. Qualitatively, the model correctly flipped "Expansion" labels to "Contingency" (Cause) labels by successfully leveraging the causal link between words like "worth" and "support."
Conclusion
The paper demonstrates that you don't necessarily need more discourse-labeled data to improve parsing; rather, you need to bridge the gap between limited training labels and the vast world of external commonsense. While successful, the system is limited by noise in external inputs and mistakes made by the CoreNLP coreference resolver. Future work involves identifying more diverse types of commonsense knowledge, such as antonyms, to help the model better distinguish "Contrast" relations.
Summary updated