DALE: A Word Sense Disambiguation System for Biomedical Documents Trained using Automatically Labeled Examples

Judita Preiss, Mark StevensonNAACL 2013

TL;DR

This paper introduces DALE, a novel Word Sense Disambiguation (WSD) system for biomedical documents that leverages automatically generated training examples. By using the UMLS Metathesaurus, DALE achieves broad coverage of ambiguous terms, matching the accuracy of supervised methods while overcoming the manual annotation bottleneck.

What Problem Does This Paper Address?

Interpreting biomedical documents accurately hinges on understanding the precise meaning of terms, which often have multiple senses (e.g., "cold" can refer to a sensation or an illness). This challenge, known as Word Sense Disambiguation (WSD), is crucial for downstream tasks like information extraction and knowledge discovery. Traditional supervised WSD systems, while accurate, require extensive manual annotation of ambiguous terms, making them expensive and difficult to scale to new domains or update with evolving terminology. Unsupervised methods offer broader coverage but generally lack the accuracy of supervised approaches. This paper addresses the need for a WSD system that combines the broad coverage of unsupervised methods with the accuracy of supervised methods, specifically for the complex and specialized domain of biomedical literature.

Methods

DALE is a supervised WSD system that tackles the manual annotation bottleneck by employing automatically labeled examples. The core of DALE's approach lies in its use of the UMLS Metathesaurus as both a source of possible meanings (Concept Unique Identifiers or CUIs) and as a resource for generating training data.

Two primary methods are used to generate these automatically labeled examples:

  1. Monosemous Relatives: This method identifies terms that are consistently associated with a single meaning (monosemous) within the UMLS Metathesaurus. The strings of these monosemous terms and their related concepts are then used to search the Medline database for instances where the ambiguous term appears.
  2. Co-occurring Concepts: This approach leverages the relationships between concepts in the UMLS Metathesaurus. It identifies frequently co-occurring concepts with the target ambiguous term and uses the strings associated with these co-occurring concepts to find relevant examples in Medline.

Once labeled examples are generated, DALE converts them into feature vectors. These vectors are derived from the lemmas of content words within the same sentence as the ambiguous term, and a ±4-word window around it. A single feature vector for each CUI is created by taking the centroid of the feature vectors from its labeled examples. The WSD process then involves comparing the feature vector of an ambiguous term in a new document against the centroids of its possible CUIs, selecting the CUI with the highest cosine similarity.

The system's pipeline includes standard NLP tasks like sentence splitting, tokenization, part-of-speech tagging, and morphological analysis, with MetaMap being used for initial identification of ambiguous terms and their potential CUIs.

Datasets

The primary resource used for generating training data is the UMLS Metathesaurus. The authors generated examples for the 2010AB, 2011AA, 2011AB, and 2012AA versions of the Metathesaurus. The system is trained and evaluated on biomedical documents, with Medline being used as the corpus for retrieving automatically labeled examples.

Key Insights and Ideas

  • Automated Labeling is Key: The most significant insight is that automatically generated training examples can effectively substitute for manually labeled ones, enabling the creation of high-coverage, accurate WSD systems without prohibitive annotation costs.
  • UMLS as a Dual Resource: The UMLS Metathesaurus proves invaluable not only as a comprehensive source of biomedical concepts and their relationships but also as a foundation for generating the necessary training data.
  • Broad Coverage with Supervised Accuracy: DALE successfully bridges the gap between unsupervised (broad coverage) and supervised (high accuracy) WSD systems, offering a practical solution for real-world biomedical text processing.
  • Monosemous Relatives and Co-occurring Concepts Complementarity: The two methods for generating labeled examples offer different but complementary ways to leverage the UMLS, contributing to the system's overall robustness.

Key Results

DALE was able to generate automatically labeled examples for a very high percentage of ambiguous CUIs across different versions of the UMLS Metathesaurus: 95.2% (2010AB), 96.2% (2011AA), 96.2% (2011AB), and 98% (2012AA). The paper claims that DALE can disambiguate biomedical documents with the coverage of unsupervised approaches and the accuracy of supervised methods. While specific quantitative accuracy scores are not detailed in the provided text, the system's ability to cover nearly all ambiguous terms in the Metathesaurus is highlighted as a major achievement.

Limitations and Future Work

  • Limitations:
    • Neither of the automatic labeling techniques could generate examples for a small percentage of CUIs. However, the authors note that these CUIs did not appear in the corresponding MetaMapped version of the Medline Repository, suggesting they are rarely used in practice.
  • The paper mentions that 100 examples were generated per CUI, as this number was found to provide the best WSD performance in the absence of other likelihood information. This implies that performance might be sensitive to the number of examples.
  • Future Work:
    • The authors intend to update DALE with new versions of the UMLS Metathesaurus as they become available, indicating a commitment to maintaining the system's currency.
  • The paper describes DALE as an "online WSD system" and provides web service interfaces, suggesting ongoing development and accessibility.

Summary updated

BibTeX
@inproceedings{preiss2013dale,
  author = {Judita Preiss and Mark Stevenson},
  title = {{DALE: A Word Sense Disambiguation System for Biomedical Documents Trained using Automatically Labeled Examples}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
}
Figure