Cross-Genre and Cross-Domain Detection of Semantic Uncertainty

György Szarvas, Veronika Vincze, Richárd Farkas, György Móra, Iryna GurevychComput. Linguistics 2012 · DOI

Abstract

Distinguishing between factual and uncertain statements—known as hedging—is essential for robust information extraction in Natural Language Processing (NLP). However, progress has been hindered by inconsistent taxonomies and the high cost of domain-specific annotation. This paper introduces a unified taxonomy for semantic uncertainty and demonstrates that domain adaptation techniques, specifically feature augmentation, can significantly reduce the manual effort required to build detectors for new fields. By harmonizing disparate datasets across biological, encyclopedic, and journalistic domains, the authors show that models can achieve high performance with a 66% reduction in required in-domain training data.

Introduction

In NLP, identifying "uncertain" statements is crucial for tasks ranging from medical record classification to fact-checking. However, uncertainty is highly context-dependent. For example, the word "evaluate" might signal uncertainty in a biological context (meaning "to judge") but represent a factual operation in mathematics (meaning "to calculate").

The field faces two primary challenges:

  1. Taxonomy Mismatch: Historically, research groups have used varying definitions and categories for uncertainty, preventing the combination of datasets.
  2. Domain Sensitivity: Uncertainty cues vary significantly across genres (e.g., news vs. scientific abstracts) and domains (e.g., biology vs. politics). Creating new detectors for every niche field is prohibitively expensive due to annotation costs.

A Unified Taxonomy

To address the lack of standardization, the authors propose a unified subcategorization of semantic uncertainty. This hierarchy organizes uncertainty into distinct types based on their linguistic function.

Hierarchical taxonomy of semantic uncertainty.

The primary categories include:

  • Epistemic: Expressing a lack of knowledge (e.g., "It may be raining").
  • Doxastic: Expressing beliefs or opinions (e.g., "He believes the Earth is flat").
  • Investigation: Propositions currently under examination (e.g., "We investigated the role of...").
  • Condition: Propositions dependent on another event (e.g., "If it rains...").

Uncertainty Cues and Features

The detection model utilizes Conditional Random Fields (CRF), a sequence labeling approach that considers the context of neighboring words. Identifying these categories relies on specific linguistic cues across different parts of speech.

Linguistic cues categorized by uncertainty type.

For cross-domain application, the authors employed a feature augmentation technique called "Easy Adapt." This method creates three versions of every feature: a "general" version shared across all domains, and two "domain-specific" versions. This allows the algorithm to distinguish between universal uncertainty cues and those unique to a specific field.

Experimental Setup and Datasets

The researchers normalized and re-annotated three major existing corpora to fit the new four-category taxonomy:

  • BioScope: Biological and clinical texts (scientific papers and abstracts).
  • WikiWeasel: Wikipedia articles (encyclopedic genre).
  • FactBank: Newswire texts (journalistic genre).

This effort resulted in a harmonized dataset of over 43,000 sentences, providing a robust benchmark for cross-domain experiments.

Key Insights and Results

The study yielded several critical findings regarding the efficiency of domain adaptation:

  • The "1k vs 3k" Rule: Using domain adaptation with only 1,000 annotated sentences in a new domain matches the performance of an in-domain model trained on 3,000 sentences. This represents a 66% reduction in human annotation labor.
  • Genre vs. Domain: The "genre" (writing style) often influences uncertainty more than the "domain" (topic). For instance, scientific abstracts are dense with "Investigation" cues, whereas full papers contain more "Conditions."
  • Disambiguation: Effective detection relies on disambiguating common words. Domain adaptation helps the model learn, for example, that "or" is a frequent uncertainty cue in biology but rarely one in news.
  • Performance: Simple cross-training achieved a 60–70% F-score, while domain adaptation with a small target dataset boosted performance to 70–80%.

Limitations and Future Work

The current experiments excluded "Dynamic Modality" (expressions of ability or permission) due to limited annotated data. Additionally, the focus remained on identifying lexical cues rather than "scope resolution"—determining exactly which part of a sentence the uncertainty modifies. Future research may integrate these domain adaptation techniques with weakly supervised methods to further lower the barrier for building specialized NLP tools.

Summary updated

BibTeX
@article{szarvas2012cross,
  author = {György Szarvas and Veronika Vincze and Richárd Farkas and György Móra and Iryna Gurevych},
  title = {{Cross-Genre and Cross-Domain Detection of Semantic Uncertainty}},
  journal = {Comput. Linguistics},
  year = {2012},
  doi = {10.1162/coli\_a\_00098},
}
Figure