TL;DR
This paper introduces "Open Mind Word Expert," a novel web-based system that leverages active learning and crowdsourcing to build large, high-quality sense-tagged corpora for Word Sense Disambiguation (WSD). By engaging the general public, the system aims to overcome the prohibitive cost and scarcity of traditional WSD datasets, paving the way for more robust WSD systems.
What Problem Does This Paper Address?
Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that aims to identify the correct meaning of a word in a given context. The performance of WSD systems, particularly supervised ones, is heavily reliant on the availability of large, semantically annotated corpora. However, creating these corpora is an extremely expensive and time-consuming process, typically requiring trained lexicographers. This has resulted in a scarcity of such datasets, limiting the development and evaluation of WSD systems. Existing datasets, while valuable, are often limited in size and scope, making it difficult to train WSD models that generalize well across a wide range of words and contexts.
Methods
The core of the paper is the "Open Mind Word Expert" system. It employs an active learning strategy to efficiently collect sense-tagged data. Here's how it works:
- Initial Tagging: Users are presented with sentences containing an ambiguous word and asked to select the correct WordNet sense from a dropdown list. Initially, sentences are drawn from existing corpora.
- Active Learning (Selective Sampling): After a small initial corpus is built, the system uses two independent classifiers (STAFS and COBALT, described in more detail below) to tag a larger pool of unlabeled data. The system then identifies sentences where these classifiers disagree. These "hard-to-tag" examples are considered the most informative and are presented to human users for annotation. This focuses human effort on the most challenging cases, maximizing the learning efficiency.
- Crowdsourcing: The system is web-based, allowing anyone to contribute. It aims to make the process engaging, almost "game-like," to encourage widespread participation.
- Quality Assurance: To ensure data quality, the system implements:
- Redundant Tagging: Each item is tagged by multiple users.
- Contributor-Specific Rules: Anonymous contributors are prevented from tagging items already tagged by users from the same IP address. Logged-in contributors are prevented from re-tagging items they've already tagged.
- Future Plans: The authors mention plans for checking session agreement and potentially using gold-standard data to assess individual contributor precision.
The two classifiers used for active learning are:
- STAFS (Semantic Tagger with Active Feature Selection): An instance-based learning algorithm that builds a separate feature space for each word and uses an iterative search to select the most informative features.
- COBALT (COnstraint-BAsed Language Tagger): Treats examples as soft constraints on word senses, using WordNet glosses and relations to identify the sense that minimizes constraint violations.
Datasets
- Initial Data Sources: The system draws its initial sentences from:
- The Penn Treebank corpus.
- The Los Angeles Times collection (provided during TREC conferences).
- Open Mind Common Sense (a collection of ~400,000 commonsense assertions).
- WordNet: Used as the sense inventory for tagging.
- SENSEVAL Datasets: Used as initial training data for the classifiers in the active learning process, particularly for words participating in SENSEVAL competitions.
- New Benchmark: The paper proposes using the corpus created by Open Mind Word Expert for a task in SENSEVAL-3. A test corpus will also be created by trained human taggers.
Key Insights and Ideas
- Crowdsourcing for WSD is Viable: The paper demonstrates that leveraging the general public via the web can be a cost-effective and scalable way to build large WSD corpora, potentially outperforming traditional methods in terms of volume and cost.
- Active Learning is Crucial for Efficiency: By focusing human annotation efforts on "hard-to-tag" examples identified by disagreeing classifiers, the system significantly reduces the annotation burden while maximizing the learning gain.
- Orthogonal Classifiers Improve Active Learning: The low agreement (54.96%) between STAFS and COBALT suggests their approaches are complementary. This orthogonality is beneficial for active learning, as disagreements highlight truly ambiguous or difficult cases.
- "Game-like" Engagement is Key: The authors emphasize that making the contribution process fun and rewarding is essential for sustained user participation and data collection. Features like tracking contributions and a "Hall of Fame" are designed to achieve this.
Key Results
- During the first fifty days of activity, the system collected approximately 26,000 tagged examples without significant publicity efforts.
- The two classifiers, STAFS and COBALT, agreed on the correct sense in 54.96% of cases on the SENSEVAL noun dataset.
- When both classifiers agreed, their combined accuracy was 82.5%, which is close to the inter-annotator agreement for SENSEVAL-2 nouns (85.5%).
- The authors estimate that a high-accuracy, domain-independent WSD system might require about 3.2 million sense-tagged words, a scale that Open Mind Word Expert aims to achieve.
Limitations and Future Work
- Initial Agreement Rate: The authors acknowledge that the initial inter-tagger agreement might be lower than in previous SENSEVAL evaluations because they are filtering out multi-word expressions, which often represent "easy cases."
- Quality Control: While initial quality assurance measures are in place, the paper mentions plans for more robust methods, including checking session agreement and using gold-standard data to assess individual contributor precision.
- Future Work:
- Replacing the "two tags per item" scheme with a "tag until at least two tags agree" scheme.
- Enriching the set of possible senses with similar meanings derived from apriori mappings or confusion matrices.
- Exploring auto-generated sense-tagged data using bootstrapping algorithms, with web contributors helping to create initial seeds and control quality.
- Expanding the system to include words with other parts of speech.
- Allowing contributors to specify more than one sense for a word.
Summary updated