Resolving Speculation: MaxEnt Cue Classification and Dependency-Based Scope Rules

Erik Velldal, Lilja Øvrelid, Stephan OepenCoNLL Shared Task 2010 · OpenReview

Abstract: Erik Velldal, Lilja Øvrelid, Stephan Oepen. Proceedings of the Fourteenth Conference on Computational Natural Language Learning – Shared Task.

2010.

Introduction

In scientific literature, distinguishing between established facts and speculative hypotheses is a fundamental requirement for automated information extraction. This process, known as hedge resolution, involves two primary challenges: identifying the specific words or phrases that signal speculation (cue detection) and determining the exact span of the sentence affected by that uncertainty (scope resolution). For instance, in the sentence "It is possible that the protein binds to DNA," the cue is "possible," and its scope is "that the protein binds to DNA."

This paper details a high-performing hybrid system developed for the CoNLL-2010 Shared Task, which focuses on biomedical text. The authors propose a pipeline that combines machine learning for cue identification with a rule-based approach for scope resolution, leveraging deep linguistic features derived from dependency parsing.

Hybrid System Architecture

The authors employed a two-level hybrid architecture designed to maximize the strengths of both statistical and symbolic methods.

Cue Identification via Maximum Entropy

For the first stage, a binary Maximum Entropy (MaxEnt) classifier was built to label individual tokens as "cue" or "non-cue." While simple n-gram baselines proved surprisingly effective, the model was significantly enhanced through parser stacking. This technique involved a data-driven parser (MaltParser) that incorporated predictions from a large-scale, hand-crafted Lexical Functional Grammar (LFG). This provided the classifier with "deep" linguistic features, such as subcategorization frames and grammatical voice, which are often missing from shallower models.

Rule-Based Scope Resolution

Rather than training a second machine learning model for scope, the authors developed a set of heuristic rules that operate on the dependency trees generated during the parsing stage. These rules define the boundaries of uncertainty based on the grammatical relationship between the cue and other sentence elements. For example, if a cue is a modal verb like may, the rules identify its subject and lexical verb dependents to define the scope.

Sample dependency representation showing the linguistic features (POS, HEAD, DEPREL) used for scope rule application.

Data and Preprocessing

The system was trained and evaluated on the BioScope corpus, which includes over 14,000 sentences from biomedical abstracts and full-text articles. A critical component of the pipeline was a custom finite-state tokenizer. The authors found that standard tools often struggled with the specialized nomenclature of biomedical text (e.g., chemical names like "Ca(2+)" or LaTeX-style dashes). By improving tokenization accuracy, they ensured that downstream PoS tagging and parsing were more reliable.

Results and Key Insights

The system was remarkably successful in the CoNLL-2010 competition, achieving the highest average rank across both sub-tasks. The official results are summarized below:

Official performance metrics for Task 1 (Uncertainty Detection) and Task 2 (Scope Resolution).

Several key insights emerged from the development and evaluation of the system:

  • Lexical Dominance: For cue identification, simple lexical information (n-grams) is the most powerful predictor. While deep dependency features improved performance, the gains showed diminishing returns once a robust lexical model was established.
  • The Utility of "Deep" Features: Features like grammatical voice (active vs. passive) and subcategorization were vital for the scope resolution rules. Knowing a verb is passive, for instance, allows the system to correctly include the subject within the scope of speculation.
  • Domain Sensitivity: The models were highly sensitive to text type. Performance on biomedical abstracts did not generalize perfectly to clinical reports, highlighting the need for domain-specific training data.

Conclusion and Future Work

The hybrid approach of combining MaxEnt classification with dependency-based rules proved to be a robust solution for hedge resolution in scientific text. While the system achieved top-tier results, the authors noted a performance drop on held-out test data, suggesting potential overfitting to training constructions. Future work may involve integrating constituent-based rules (phrase structure) or employing a statistical ranker to choose between multiple scope candidates, potentially further refining the precision of the scope boundaries.

Summary updated

BibTeX
@inproceedings{velldal2010resolving,
  author = {Erik Velldal and Lilja Øvrelid and Stephan Oepen},
  title = {{Resolving Speculation: MaxEnt Cue Classification and Dependency-Based Scope Rules}},
  booktitle = {CoNLL Shared Task},
  year = {2010},
  url = {https://openreview.net/forum?id=2hLkTONadiC},
}
Figure