Inducing Domain-Specific Semantic Class Taggers from (Almost) Nothing

Ruihong Huang, Ellen RiloffACL 2010 · OpenReview

Abstract

Semantic class tagging is a foundational NLP task, yet its application in specialized domains like veterinary medicine is often hindered by the prohibitive cost of manual annotation. Ruihong Huang and Ellen Riloff present a novel bootstrapping architecture that induces domain-specific taggers from unannotated text and a small set of seed words. Their approach utilizes a "blindfolded" contextual classifier and a competitive cross-category bootstrapping mechanism to achieve an F-score of 76.6, significantly outperforming supervised baselines trained on limited data.

Introduction

In specialized fields such as law or medicine, general-purpose NLP tools often fail due to dense jargon and domain-specific word senses. For instance, in a veterinary context, "doxy" might refer to a dachshund or the antibiotic doxycycline. Creating high-quality taggers for these domains typically requires massive amounts of expert-annotated data. This paper addresses the bottleneck by proposing a method to build accurate semantic taggers from "(almost) nothing"—leveraging raw text and just 10 seed words per category.

Methodology: Bootstrapping with Competition

The authors propose a two-phase bootstrapping architecture designed to prevent the common pitfall of "semantic drift," where a model gradually begins to include unrelated terms in its learned categories.

Phase 1: The Strictly Contextual Classifier

To initiate the process, the system labels instances of seed words in the corpus. However, to prevent the model from simply memorizing these seeds, the authors "mask" the target word in the feature vector. This "blindfolding" forces the classifier to look only at the surrounding contextual patterns (e.g., "diagnosed with [X]") rather than the words themselves, facilitating better generalization.

Phase 2: Cross-Category Bootstrapping

The core of the system involves training a suite of binary classifiers simultaneously—one for each target category (e.g., ANIMAL, DRUG, DISEASE). In each iteration, high-confidence positive predictions from one category are used as negative training examples for all others. This competitive environment acts as a natural regularizer, keeping the classifiers in check and preventing them from drifting into each other's semantic space.

Architecture of the cross-category bootstrapping system.

Enhancements

  • One-Semantic-Class-Per-Discourse (OSCPD): If a word is confidently tagged within a specific forum thread, the system applies that tag to all other instances of the same head noun in that thread.
  • Dynamic Semantic Features: Confidently tagged words are injected as new features for their neighbors in subsequent iterations.
  • Sliding Confidence Threshold: The system begins with a strict threshold for labeling new data, gradually lowering it as the process exhausts high-confidence examples.

Experimental Evaluation

The system was evaluated using an unannotated training corpus of 4,629 message board threads from the Veterinary Information Network (VIN). A test set of 100 posts was manually annotated by domain experts (κ = 0.80) across six categories: ANIMAL, DISEASE/SYMPTOM, DRUG, TEST, HUMAN, and OTHER.

Key Results

The final system (Cross-Category + OSCPD + Semantic Features) achieved an average F-score of 76.6 (75% recall, 78% precision). As shown in the learning curves, the cross-category approach significantly outperformed independent self-training, which stalled early with an F-score of 67.4.

F-measure performance over iterations for various bootstrapping configurations.

Remarkably, the bootstrapped model outperformed a supervised model trained on the expert-annotated test set, which only reached a 60.7 F-score. This demonstrates that for specialized domains, a large volume of unlabeled data combined with intelligent bootstrapping can be more effective than a small amount of expensive labeled data.

Conclusion

This research demonstrates that domain-specific semantic taggers can be effectively induced with minimal human intervention. By forcing generalization through contextual masking and maintaining precision through cross-category competition, the proposed system provides a scalable solution for extracting structured information from specialized text corpora. Future work may involve integrating these taggers into broader NLP pipelines for tasks like coreference resolution or event extraction.

Summary updated

BibTeX
@inproceedings{huang2010inducing,
  author = {Ruihong Huang and Ellen Riloff},
  title = {{Inducing Domain-Specific Semantic Class Taggers from (Almost) Nothing}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2010},
  url = {https://openreview.net/forum?id=lvZilIivjEt},
}
Figure