TL;DR
This paper tackles the challenge of acquiring high-quality annotations for hedging language, which expresses uncertainty or ambiguity. They introduce a novel crowdsourcing method that reframes hedge identification as a word sense disambiguation task, leading to a new, large-scale corpus of informal forum posts annotated for hedging with impressive accuracy.
What Problem Does This Paper Address?
Hedging is a crucial linguistic phenomenon used to express uncertainty, soften statements, or distance speakers from their propositions. Identifying hedging is vital for understanding nuances in communication, social dynamics, and information extraction. However, studying hedging has been hampered by a lack of readily available, high-quality annotated datasets, especially for informal text like online forums. Existing corpora are often domain-specific (e.g., biomedical texts) and annotated by expensive expert linguists, limiting the scope and scale of hedging research. This paper aims to bridge this gap by developing an efficient and scalable method for collecting hedging annotations.
Methods
The core innovation of this paper lies in reframing the hedge identification task as a word sense disambiguation (WSD) problem. Instead of asking annotators to directly identify "hedges," they present a sentence with a potential hedge word highlighted and offer two definitions: one for the hedging usage and one for a non-hedging usage. Annotators then choose the definition that best matches the word's meaning in the given sentence. This approach simplifies the task for non-expert crowdsourced workers.
They first compiled a dictionary of 117 potential hedge words and phrases, drawing from existing resources and expanding it with terms relevant to informal text. For each term, they carefully crafted distinct hedging and non-hedging definitions with example sentences. These definitions were iteratively refined based on pilot studies with real-world users to ensure clarity and minimize ambiguity. The crowdsourcing was then performed using Amazon Mechanical Turk (AMT), where each potential hedge instance was judged by multiple workers.
Datasets
The primary dataset used in this work is a newly constructed corpus of informal discussion forum posts. These posts were sourced from the 2014 Deft Committed Belief Corpora (LDC2014E55, LDC2014E106, and LDC2014E125), selected for their informal nature and focus on dynamic events or personal anecdotes. This corpus contains 20,683 annotated potential hedge terms.
The authors also reference existing datasets and tasks:
- CoNLL-2010 shared task: This task focused on detecting hedges and their scope, using the BioScope corpus (biomedical abstracts) and a Wikipedia corpus annotated for "weasel words."
- BioScope corpus (Vincze et al., 2008): Used in the CoNLL-2010 task, it's domain-specific.
- Wikipedia corpus: Also used in CoNLL-2010, but its definition of "weasel words" differs slightly from hedging.
The new corpus introduced in this paper is notable for its focus on informal language, a genre previously lacking in hedging annotations.
Key Insights and Ideas
- Reframing as WSD is effective: The paper demonstrates that by simplifying the task to word sense disambiguation, it's possible to leverage crowdsourcing for acquiring high-quality hedging annotations, even from non-expert annotators.
- Informal language is rich in hedging: The creation of a new corpus from forum posts highlights the prevalence and importance of hedging in everyday online communication.
- Annotation quality can be very high: The crowdsourced annotations achieved impressive agreement rates, suggesting that the WSD framing and careful definition crafting were successful.
- Agreement varies with ambiguity: The study found that agreement among annotators was higher for terms that are consistently used as hedges or non-hedges, and lower for more ambiguous terms, which is an intuitive finding.
Key Results
- High Inter-Annotator Agreement: The crowdsourced judgments achieved an overall agreement of 92.89%, with a Fleiss' Kappa of 0.751.
- High Accuracy against Gold Standard: When comparing a subset of the crowdsourced annotations to an expert-annotated gold standard, the accuracy was 96.65%.
- New Corpus: A corpus of 20,683 annotated potential hedge terms from informal forum posts was created.
- Expert Agreement: Inter-annotator agreement among the paper's authors on a subset was 94.73% (Cohen's Kappa = 0.857).
Limitations and Future Work
- Data Imbalance: The resulting corpus is highly unbalanced, with some hedge terms appearing far more frequently than others.
- Ambiguous Terms: For certain highly ambiguous terms, the simple binary choice of hedging vs. non-hedging definitions might not be sufficient. The authors suggest that for such terms, a more comprehensive set of definitions or a direct question about the speaker's intent (uncertainty/lack of commitment) might be more effective.
- Future Directions: The authors hope to make their corpus generally available through the LDC. Further work could explore more nuanced annotation schemes for highly ambiguous terms or investigate how these annotations can be used to improve downstream NLP tasks.
Summary updated