Using Hedge Detection to Improve Committed Belief Tagging

Morgan Ulinski, Seth Benjamin, Julia HirschbergNAACL 2018

TL;DR

This paper introduces a novel rule-based method for detecting hedges in informal text, specifically forum posts. By integrating these hedge features into a committed belief tagging system, the authors demonstrate significant improvements in identifying nuanced expressions of belief, particularly for reported and non-committed beliefs.

What Problem Does This Paper Address?

Understanding the level of commitment a speaker or writer has towards a proposition (committed belief tagging) is crucial for many natural language understanding tasks. However, this is complicated by the use of "hedges" – words or phrases that introduce ambiguity or uncertainty. Identifying these hedges is challenging because many hedge terms are polysemous, meaning they can be used in both hedging and non-hedging ways (e.g., "around" in "around a hundred pounds" vs. "turned around"). Existing research on hedge detection has largely focused on formal domains like academic writing, and there's a lack of annotated corpora for informal genres like online forums. This paper tackles the problem of accurately detecting hedges in informal text and leveraging this detection to enhance committed belief tagging.

Methods

The authors propose a two-stage approach:

  1. Hedge Identification:
    • Dictionary-Based: A baseline approach uses a dictionary of 117 potential hedge terms (compiled from existing datasets and WordNet) to identify hedges by simply looking up word lemmas.
  2. Rule-Based: To address the ambiguity of hedge terms, they developed a more sophisticated rule-based system. This system uses manually constructed rules based on context, part-of-speech, and dependency information to disambiguate whether a potential hedge term is actually being used as a hedge. For terms without specific rules, they are classified as hedges by default.
  3. Committed Belief Tagger Enhancement:
    • The identified hedge features (token, lemma, and type – relational or propositional) are incorporated into an existing committed belief tagger. This tagger uses a quadratic kernel SVM and relies on lexical and syntactic features.
  4. The hedge features are added at multiple levels: as word features for the current token, as dependency features for related tokens (parent, sibling, ancestor), and as a sentence-level feature indicating if any hedge exists in the sentence.

Datasets

  • Darpa DEFT Committed Belief Corpus (Release No. LDC2014E55): This corpus, consisting of English discussion forum data, is used for training and evaluating the committed belief tagger. The authors note that their baseline results differ slightly from prior work due to variations in training/evaluation datasets.
  • CoNLL-2010 Shared Task Data: The initial list of hedge terms was partly derived from this task.
  • WordNet: Used to expand the initial list of hedge terms with synonyms.
  • Amazon Mechanical Turk (AMT): The authors plan to use AMT to construct a new corpus of forum posts labeled with hedge information, which will be used to evaluate their rule-based detector and train machine learning classifiers in future work.

Key Insights and Ideas

  • Rule-based disambiguation is crucial: Simply relying on a dictionary of potential hedge terms is insufficient due to ambiguity. Manually crafted rules that consider linguistic context significantly improve hedge detection accuracy.
  • Hedges are vital for nuanced belief expression: The impact of hedge features is most pronounced in detecting "Reported Belief" (ROB) and "Non-Committed Belief" (NCB), which represent less certain or attributed beliefs. This highlights the importance of hedges in conveying subtle stances.
  • Informal text presents unique challenges: The need for a rule-based system tailored to informal genres like forum posts underscores the limitations of applying models trained on formal text directly.

Key Results

The paper presents results in Table 3, comparing three systems:

  • (a) No hedge features (Baseline): Overall F-measure of 67.52.
  • (b) Dictionary-based hedge detection: Overall F-measure of 67.95. This shows a modest improvement over the baseline.
  • (c) Rule-based hedge detection: Overall F-measure of 68.07. This represents the best performance.

While the overall F-measure improvements are relatively small (0.43 for dictionary-based, 0.55 for rule-based), the gains are more significant for specific belief types:

  • ROB: F-score increased from 23.29 (baseline) to 27.43 (rule-based).
  • NCB: F-score increased from 23.66 (baseline) to 30.43 (rule-based).

These specific improvements, though on smaller subsets of the data, are substantial and demonstrate the value of accurate hedge detection for understanding nuanced beliefs.

Limitations and Future Work

  • Limitations of Rule-Based Systems: The authors acknowledge that defining simple rules to disambiguate all hedge uses can be challenging, especially in complex cases.
  • Future Work:
    • AMT Corpus: Constructing a larger, annotated corpus of forum posts using Amazon Mechanical Turk to further evaluate the rule-based detector and train machine learning classifiers.
  • Machine Learning Classifiers: Using the new AMT corpus to train ML classifiers for hedge detection, potentially surpassing the performance of rule-based systems.
  • Improved Disambiguation: Continuing to refine the rule-based system and explore more sophisticated disambiguation techniques.

Summary updated

BibTeX
@inproceedings{ulinski2018using,
  author = {Morgan Ulinski and Seth Benjamin and Julia Hirschberg},
  title = {{Using Hedge Detection to Improve Committed Belief Tagging}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2018},
}
Figure