Kernel Methods for Word Sense Disambiguation and Acronym Expansion

Mahesh Joshi, Ted Pedersen, Richard Maclin, Serguei V. S. PakhomovAAAI 2006

Abstract

Supervised machine learning models for Word Sense Disambiguation (WSD) often face a "knowledge acquisition bottleneck" due to the high cost of manual data labeling. This paper investigates the application of kernel methods within Support Vector Machines (SVMs) to mitigate this issue. By integrating unlabeled text and structured ontologies through Latent Semantic and Domain Kernels, the authors demonstrate significant accuracy improvements in both general medical WSD and clinical acronym expansion.

Introduction

Word Sense Disambiguation (WSD) is the process of identifying which sense of a word is used in a given context—for instance, distinguishing between "bank" as a financial institution and "bank" as a river edge. While supervised learning has proven effective for this task, such models require extensive labeled training sets. In specialized domains like medicine, where acronyms such as "MS" (e.g., Multiple Sclerosis vs. Mitral Stenosis) are context-dependent and abundant, the scarcity of labeled data creates a significant barrier to performance.

This research addresses the knowledge acquisition bottleneck by leveraging "free" resources: large-scale unlabeled text and existing structured ontologies like WordNet and the Unified Medical Language System (UMLS).

Methodology: Kernel-Based Learning

The authors utilize Support Vector Machines (SVMs), focusing on kernel methods to project data into high-dimensional feature spaces where semantic similarities can be more effectively calculated.

Latent Semantic and Domain Kernels

To move beyond simple bag-of-words features, the study employs:

  • Latent Semantic Kernels (LSK): These kernels utilize unlabeled text to identify underlying patterns of word usage. This allows the model to recognize semantic relationships between words that may not co-occur in the limited labeled training set.
  • Domain Kernels: These incorporate domain-specific knowledge to help the classifier distinguish between senses based on the broader subject matter.

Feature Engineering in Clinical Contexts

For clinical notes, the authors moved beyond textual features to include rich metadata:

  • Patient Metadata: Demographic information such as gender.
  • Contextual Metadata: The specific section of the clinical note and the hospital department of origin.
  • Flexible Window Approach: Instead of a fixed-width context window, the model identifies the most significant content words regardless of their exact distance from the target word.

Experimental Datasets

The methodology was evaluated on two distinct datasets:

  1. NLM WSD Collection: Comprising 50 ambiguous terms from the U.S. National Library of Medicine, mapped to UMLS concepts.
  2. Mayo Clinic Clinical Notes: An 8-acronym dataset derived from real-world clinical documentation, used to test acronym expansion capabilities.

Results and Analysis

The experimental results validate the efficacy of kernel-based approaches in data-scarce environments.

WSD Performance

On the NLM medical terms, a standard SVM baseline achieved an average accuracy of 76.26%. The introduction of kernels based on unlabeled text improved accuracy on a subset of challenging words from 71.64% to 74.62%. Notably, the word "mosaic" saw a 12-percentage-point increase in accuracy despite having the smallest training set, illustrating the power of unlabeled data in providing "connective tissue" for rare terms.

Acronym Expansion

In the clinical notes dataset, the use of a linear kernel combined with clinical-specific features (like department metadata) yielded a high average accuracy of 93.26%. This underscores the importance of non-textual signals in specialized environments; for example, a note from a Cardiology department provides a strong prior probability that "MS" refers to Mitral Stenosis rather than Multiple Sclerosis.

Discussion: Kernel Additivity

A key insight from the paper is the modularity of kernel methods. The authors propose "kernel additivity," where individual kernels representing different knowledge sources (e.g., one for unlabeled text, another for ontologies) can be summed to create a more robust additive kernel. This allows for a flexible integration of diverse data types without requiring a complete redesign of the feature space.

Conclusion and Future Work

The study demonstrates that kernel methods effectively utilize unlabeled data and metadata to overcome the knowledge acquisition bottleneck. While the relationship between the volume of unlabeled data and accuracy gains requires further investigation, the results suggest a promising path for semi-supervised learning in NLP.

Future research will focus on:

  • Integrating structured knowledge directly from WordNet and UMLS into specific kernels.
  • Developing dynamic weighting schemes to automatically prioritize the most relevant knowledge source for a given context.
  • Refining the flexible window approach to better capture long-distance linguistic dependencies.

Summary updated

BibTeX
@inproceedings{joshi2006kernel,
  author = {Mahesh Joshi and Ted Pedersen and Richard Maclin and Serguei V. S. Pakhomov},
  title = {{Kernel Methods for Word Sense Disambiguation and Acronym Expansion}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2006},
}
Figure