Revisiting Syntax-Based Approach in Negation Scope Resolution

Asahi Yoshida, Yoshihide Kato, Shigeki MatsubaraACL 2023

TL;DR

This paper revisits traditional syntax-based methods for negation scope resolution, demonstrating that by using modern, high-accuracy parsers and slightly refining heuristic rules, these methods can achieve performance comparable to state-of-the-art end-to-end neural networks. This work highlights the continued relevance of syntactic structure for interpretable NLP tasks.

What Problem Does This Paper Address?

Negation scope resolution is a crucial NLP task that involves identifying which part of a sentence is affected by a negation cue (like "not"). Traditionally, this has been tackled using syntactic parsing to understand sentence structure. However, recent advances have favored end-to-end deep learning models that often bypass explicit syntactic analysis, claiming superior performance. This paper questions this trend, proposing that the decline in performance of syntax-based methods might be due to outdated parsing technology rather than an inherent limitation of the approach itself. They aim to demonstrate that with contemporary parsing tools, syntax-based methods can regain their competitive edge and offer greater interpretability.

Methods

The core of this work is a re-evaluation of the syntax-based approach proposed by Read et al. (2012). The authors make two primary modifications:

  1. Parser Replacement: Instead of using the older reranking parser from the *SEM2012 shared task, they employ state-of-the-art parsers: the Berkeley Neural Parser and the Attach Juxtapose Parser. These parsers are known for their high accuracy on the Penn Treebank benchmark.
  2. Heuristic Rule Modification: They slightly adjust the heuristic rules used for selecting the negation scope from the parse tree. Specifically, they modify a rule for handling relative clauses to better align with how modern parsers represent such structures and adjust rules for scope adjustment to better handle discourse markers and sentential adverbs.

The overall process involves parsing the sentence, identifying candidate scope constituents that dominate the negation cue, selecting the most likely scope constituent using modified heuristics, and finally adjusting the selected scope by removing certain elements.

Datasets

The primary dataset used for evaluation is the Conan Doyle dataset, also known as the *SEM2012 shared task dataset. This dataset includes sentences annotated for negation cues and their scopes. It is divided into training, development, and evaluation sets. The authors use the evaluation data for their experiments.

Key Insights and Ideas

  • Parsers Matter: The performance of syntax-based NLP methods is highly dependent on the quality of the underlying parser. Using modern, high-accuracy parsers significantly boosts the effectiveness of syntax-based negation scope resolution.
  • Syntax Still Relevant: Despite the rise of end-to-end deep learning, explicit syntactic structure remains a valuable feature for tasks like negation scope resolution, especially when interpretability is desired.
  • Heuristics Can Be Refined: The heuristic rules used in older syntax-based methods can be effectively updated to work with modern parsing outputs and linguistic conventions, leading to improved performance.
  • Interpretability Advantage: Syntax-based methods offer a clear advantage in interpretability compared to black-box neural models, allowing researchers to understand why a particular scope was chosen.

Key Results

The paper reports significant improvements in negation scope resolution performance by using state-of-the-art parsers and modified heuristics.

  • Scope-level F1: With the Attach Juxtapose Parser and modified rules, the system achieved a scope-level F1 score of 85.13%, which is comparable to state-of-the-art end-to-end neural methods.
  • Token-level F1: The same configuration yielded a token-level F1 score of 92.61%, also competitive with recent deep learning approaches.
  • Comparison to Prior Work: The modified syntax-based approach, when tested with the Attach Juxtapose Parser, achieved a token-level F1 score of 91.74% (including punctuation), only slightly behind the best-performing deep learning method (Khandelwal and Sawant, 2020) at 92.36%.

Limitations and Future Work

Limitations:

  • The authors note that the sentential adverb list used in Read et al. (2012) for scope adjustment was not publicly available, so they had to implement their own generalized rule for removing comma-delimited adverbial phrases or interjections. This might introduce differences in how discourse markers are handled.
  • The evaluation was performed using gold cues, meaning the negation cues themselves were provided and not detected by the system.

Future Work:

  • The authors plan to integrate a tree-based neural model into the constituent selection process to further enhance performance.
  • They suggest exploring the application of their syntax-based approach to other linguistic phenomena, such as speculation or quantification.

Summary updated

BibTeX
@inproceedings{yoshida2023revisiting,
  author = {Asahi Yoshida and Yoshihide Kato and Shigeki Matsubara},
  title = {{Revisiting Syntax-Based Approach in Negation Scope Resolution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2023},
}

Loading related content...

Figure