The Virginia Tech System at CoNLL-2016 Shared Task on Shallow Discourse Parsing

Prashant Chandrasekar, Xuan Zhang, Saurabh Chakravarty, Arijit Ray, John Krulick, Alla RozovskayaCoNLL Shared Task 2016 · OpenReview

TL;DR

This paper presents the Virginia Tech system for the CoNLL-2016 Shallow Discourse Parsing Shared Task. Their pipeline-based system builds on previous work but introduces novel features for explicit sense classification and argument boundary identification, achieving a respectable F1 score of 20.27% on the blind test set.

What Problem Does This Paper Address?

Discourse parsing aims to understand how sentences and clauses in a text relate to each other, going beyond simple sentence structure to reveal the underlying logical and rhetorical connections. This is crucial for tasks like text summarization, question answering, and machine translation, where understanding the flow of information is paramount. The CoNLL-2016 shared task specifically focused on "shallow" discourse parsing, meaning identifying explicit and implicit discourse relations, their arguments (the text spans involved), and their senses (e.g., Cause, Contrast, Expansion). The challenge lies in accurately identifying these relations, especially implicit ones where no explicit connective word (like "because" or "but") is present, and precisely determining the boundaries of the arguments involved.

Methods

The Virginia Tech system employs a pipeline architecture, where each component handles a specific sub-task. The core components include:

  • Explicit Connective Identification: A binary classifier to detect words or phrases that function as discourse connectives.
  • Argument Position Identification: For explicit relations, this determines if Arg1 is in the same sentence (SS) or a previous sentence (PS) as the connective.
  • Argument Extraction (Explicit): Separate modules for SS and PS relations to extract the text spans for Arg1 and Arg2. They leverage existing techniques but introduce novel features, including POS information and punctuation around constituents.
  • Explicit Sense Classification: A classifier to assign a sense to identified explicit relations. They introduce new features related to the connective's tokens and surrounding words.
  • Non-Explicit Relation Identification: This involves identifying candidate sentence pairs that might have an implicit relation, based on adjacency and paragraph structure.
  • Non-Explicit Sense Classification: A two-pass classifier. The first pass distinguishes EntRel (sentence-level relations) from implicit relations. The second pass, after argument boundaries are identified, refines the sense prediction.
  • Implicit Argument Extraction: Modules to identify the boundaries of Arg1 and Arg2 for implicit relations. They use features similar to explicit argument extraction but focus on dependency relations within constituents.

A key innovation is the introduction of novel features for explicit sense classification and argument extraction, aiming to improve performance over state-of-the-art systems from the previous year.

Datasets

The system was trained and tuned using data adapted from the Penn Discourse Treebank 2.0 (PDTB-2.0). Evaluation was performed on the development, test, and official blind test sets from PDTB.

Key Insights and Ideas

  • Pipeline Architecture is Effective: The paper reinforces the success of a modular, pipeline approach for discourse parsing, building upon established architectures.
  • Novel Features Boost Performance: The introduction of new features, particularly for explicit sense classification and argument extraction (PS Arg1 and PS Arg2), demonstrably improved performance, even on top of strong baseline models. This highlights the continued importance of feature engineering in NLP.
  • Two-Pass Non-Explicit Sense Classification: The strategy of using a two-pass approach for non-explicit sense classification, first separating EntRel and then refining implicit senses, is a pragmatic way to handle the complexity of these relations.
  • Importance of Contextual Features: The novel features often incorporate information about surrounding words, POS tags, and punctuation, underscoring the value of local context for disambiguating discourse relations.

Key Results

The Virginia Tech system achieved an F1 score of 20.27% on the official blind test set in the CoNLL-2016 shared task.

  • Explicit Sense Classifier: New features improved performance by almost 1 F1 point over their baseline.
  • PS Arg1 Extractor: Achieved a 5-point improvement over the baseline.
  • PS Arg2 Extractor: Showed a larger improvement of three F1 points over the baseline.
  • Implicit Arg1 Extractor: Demonstrated a significant improvement of 12 F1 points over the baseline.

Limitations and Future Work

  • No Error Propagation (EP) vs. EP: While component-level evaluations were performed without error propagation (using gold standard inputs), the overall system performance is impacted by errors cascading through the pipeline. The paper notes that "each component contributes errors."
  • Noisy Candidate Generation for Implicit Relations: The method for generating candidate sentence pairs for implicit relations is described as "quite noisy," with about 24% of pairs not participating in a discourse relation. This is identified as an area for future work.
  • Limited Comparison of Algorithms: Due to time constraints, the authors did not compare all three algorithms (Maximum Entropy, Averaged Perceptron, Naïve Bayes) on all sub-tasks, relying on prior work for algorithm selection.
  • Missed Arg1 Positions: For explicit relations, the system only considers the immediately preceding sentence for Arg1 when it's in a previous sentence (PS). About 10% of explicit relations have Arg1 occurring in sentences further back, which are missed.

Future work could focus on improving the candidate generation for implicit relations and exploring more sophisticated ways to handle Arg1 positions in explicit relations.

Summary updated

BibTeX
@inproceedings{chandrasekar2016virginia,
  author = {Prashant Chandrasekar and Xuan Zhang and Saurabh Chakravarty and Arijit Ray and John Krulick and Alla Rozovskaya},
  title = {{The Virginia Tech System at CoNLL-2016 Shared Task on Shallow Discourse Parsing}},
  booktitle = {CoNLL Shared Task},
  year = {2016},
  url = {https://openreview.net/forum?id=BW7IvI1SCEh},
}
Figure