Labeling Explicit Discourse Relations Using Pre-trained Language Models

Murathan Kurfali

TL;DR

This paper demonstrates that pre-trained language models like BERT can effectively replace complex, hand-crafted linguistic features for identifying explicit discourse relations in text. By fine-tuning BERT, the authors achieve state-of-the-art results on the PDTB 2.0 dataset, marking a significant step towards simpler and more generalizable discourse parsing systems.

What Problem Does This Paper Address?

Shallow discourse parsing (SDP) aims to identify how sentences and clauses relate to each other semantically (e.g., cause, contrast, precedence). A key sub-task is labeling explicit discourse relations, which involves finding the words that signal these connections (connectives) and the text spans they link (arguments). This task is challenging because connectives can be ambiguous, arguments can be discontinuous, and the same text span can play different roles in different relations. Historically, state-of-the-art models relied heavily on intricate, hand-engineered linguistic features, making them complex and difficult to adapt to new languages or domains. This paper investigates whether modern pre-trained language models can achieve comparable or better performance without this feature engineering.

Methods

The authors propose a simplified two-step pipeline for labeling explicit discourse relations:

  1. Connective Identification: This step identifies the lexical items that function as discourse connectives. The model treats this as an N-way token classification problem, using a BERT model with a linear layer on top. To handle multi-word connectives, they introduce distinct labels for single-token and multi-word connectives.
  2. Argument Extraction: Once a connective is identified, this step extracts the spans of text that constitute its arguments (Arg1 and Arg2). This is also framed as a token-level sequence prediction task using BERT. The model searches for arguments within a fixed window (100 words) centered around the connective. Arguments are labeled using the IOB2 format.

Crucially, the entire pipeline is built using a simple neural architecture that leverages BERT's contextual embeddings, eliminating the need for any hand-crafted linguistic features like dependency trees or part-of-speech tags.

Datasets

The primary dataset used for training and evaluation is the Penn Discourse Treebank (PDTB) 2.0. The authors follow the standard CoNLL 2015/2016 shared task split, using sections 2-21 for training, section 22 for development, and section 23 for testing.

Key Insights and Ideas

  • Pre-trained Models Replace Feature Engineering: The most significant insight is that fine-tuning pre-trained language models like BERT is sufficient to achieve state-of-the-art performance in explicit discourse relation labeling, completely obviating the need for complex, manually designed linguistic features.
  • Simplified Pipeline: The proposed two-step pipeline (connective identification followed by argument extraction) is effective and simpler than previous multi-stage approaches.
  • BERT's Power for Context: BERT's ability to capture rich contextual information is key to disambiguating connectives and identifying argument spans, even in challenging cases like discontinuous arguments.
  • Generalizability Potential: By removing feature engineering, the model becomes more directly applicable to raw text and has the potential to be extended to other languages with available data and pre-trained models.

Key Results

The paper reports state-of-the-art results on the PDTB 2.0 test set:

  • Connective Identification: Achieved an F-score of approximately 96.77%, significantly outperforming previous methods.
  • Argument Extraction: Achieved an F-score of approximately 60.12% for Arg1 and 80.50% for Arg2.
  • Full Relation Extraction (Arg1+Arg2): Achieved an F-score of approximately 53.28%, which is an 8% improvement over previous feature-independent models and competitive with or superior to many feature-based systems.
  • The model demonstrates strong performance even on relations where the second argument precedes the first, achieving a 75.6% F-score for full relation extraction in such cases.

Limitations and Future Work

  • Discontinuous Arguments: The paper acknowledges that extracting arguments in discontinuous spans, especially when they are far from the connective, remains a challenge. The current window-based approach has limitations in these cases.
  • Unseen Connectives: The test set did not contain any connectives not seen during training. Therefore, the model's generalization capabilities to entirely novel connectives are not yet demonstrated.
  • Future Work: The authors suggest further examination of generalization capabilities and extending the system to other languages. They also envision the system as a foundational step towards more high-performance, generalizable shallow discourse parsers.

Summary updated

Abstract

Labeling explicit discourse relations is one of the most challenging sub-tasks of the shallow discourse parsing where the goal is to identify the discourse connectives and the boundaries of their arguments. The state-of-the-art models achieve slightly above 45% of F-score by using hand-crafted features. The current paper investigates the efficacy of the pre-trained language models in this task. We find that the pre-trained language models, when finetuned, are powerful enough to replace the linguistic features. We evaluate our model on PDTB 2.0 and report the state-of-the-art results in extraction of the full relation. This is the first time when a model outperforms the knowledge intensive models without employing any linguistic features.
BibTeX
@inproceedings{kurfali2020labeling,
  author = {Murathan Kurfali},
  title = {{Labeling Explicit Discourse Relations Using Pre-trained Language Models}},
  booktitle = {TDS},
  year = {2020},
  url = {https://openreview.net/forum?id=q_G_0epXvhA},
}
Figure