SupWSD: A Flexible Toolkit for Supervised Word Sense Disambiguation

Simone Papandrea, Alessandro Raganato, Claudio Delli BoviEMNLP 2017

TL;DR

This paper introduces SUPWSD, a flexible and modular Java toolkit for supervised Word Sense Disambiguation (WSD). It addresses the limitations of existing WSD systems, particularly the lack of multilingual support and extensibility, by offering a state-of-the-art implementation with an easy-to-use API and a RESTful service.

What Problem Does This Paper Address?

Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Understanding, aiming to identify the correct meaning of a word in its context. While supervised WSD models generally outperform knowledge-based ones, existing state-of-the-art systems like "It Makes Sense" (IMS) are often limited to English, difficult to extend, and inefficient for large datasets. This creates a bottleneck for researchers wanting to work with multiple languages or large-scale data. The paper tackles this by providing a robust, flexible, and efficient WSD toolkit that supports multilingualism and is designed for easy extension.

Methods

SUPWSD is built around a modular, four-stage pipeline:

  1. Input Parsing: Handles various input formats (plain text, XML) and efficiently loads data using a lazy procedure to manage large files. It supports multiple parsers, allowing for easy integration of new ones.
  2. Text Preprocessing: Implements a five-step pipeline (sentence splitting, tokenization, POS tagging, lemmatization, dependency parsing) using either Stanford NLP or a hybrid approach, allowing users to customize models for different languages.
  3. Features Extraction: Constructs features from the preprocessed text, including POS tags, surrounding words, local collocations, word embeddings (with various strategies), and syntactic relations. Users can configure which features to use and set frequency cutoffs.
  4. Classification: Leverages off-the-shelf machine learning libraries like LIBLINEAR and LIBSVM to train and test WSD models. It supports WordNet and BabelNet as sense inventories, enabling the use of a Most Frequent Sense (MFS) back-off strategy.

The toolkit is designed with common Java design patterns (Factory, Observer) to ensure flexibility and extensibility.

Datasets

The paper evaluates SUPWSD on five standard all-words WSD datasets: Senseval-2, Senseval-3, SemEval-07, SemEval-13, and SemEval-15. For training, it uses SemCor and OMSTI corpora. WordNet 3.0 is used as the sense inventory.

Key Insights and Ideas

  • Bridging the Multilingual Gap: The paper highlights that while knowledge-based WSD systems are easier to scale to multiple languages, supervised models offer superior performance. SUPWSD aims to bridge this gap by providing a flexible framework that can leverage multilingual preprocessing tools.
  • Extensibility as a Core Principle: By designing SUPWSD with modularity and standard Java design patterns, the authors make it significantly easier for researchers to add new parsers, preprocessing modules, features, and classifiers, fostering community contributions.
  • Efficiency Matters: The toolkit is not only flexible but also optimized for speed. The authors demonstrate a significant speedup in both training and testing phases compared to the IMS system, making it practical for large-scale WSD tasks.
  • Unified Evaluation Framework: The use of the Raganato et al. (2017) evaluation framework provides a standardized way to compare SUPWSD against other systems across multiple datasets.

Key Results

  • Performance: SUPWSD consistently matches or slightly outperforms the original IMS implementation in terms of F-Measure across five all-words WSD datasets, demonstrating that flexibility doesn't compromise accuracy.
  • Speed: SUPWSD achieves a significant speedup, being approximately 3 times faster than IMS on the Semcor training set and almost 6 times faster on the OMSTI training set. This efficiency is crucial for practical WSD applications.

Limitations and Future Work

  • Current Limitations: The paper notes that the current version of SUPWSD relies on two specific machine learning frameworks (LIBLINEAR and LIBSVM) and two sense inventories (WordNet and BabelNet). While the framework is extensible, these are the currently implemented options.
  • Future Work: The authors suggest that the modular design allows for easy integration of new parsers, preprocessing modules, features, and classifiers. The release of the toolkit under open-source licenses (GNU GPL v3.0 for the framework, CC BY-NC-SA 3.0 for the RESTful service) encourages community contributions and further development.

Summary updated

BibTeX
@inproceedings{papandrea2017supwsd,
  author = {Simone Papandrea and Alessandro Raganato and Claudio Delli Bovi},
  title = {{SupWSD: A Flexible Toolkit for Supervised Word Sense Disambiguation}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
}
Figure