TL;DR
This paper introduces a novel, yet simpler, neural network architecture for Word Sense Disambiguation (WSD) using bidirectional LSTM cells. The proposed system achieves state-of-the-art results on English WSD tasks while being easier to train and more reproducible than previous neural approaches, thanks to the use of publicly available resources.
What Problem Does This Paper Address?
Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that aims to identify the correct meaning of a word in a given context from a predefined set of senses. For example, the word "bank" can refer to a financial institution or the side of a river. Traditional WSD methods often rely on knowledge-based resources or complex feature engineering. While neural networks have recently shown great promise in WSD, achieving top performance in evaluation campaigns, they are often complex to train and their reproducibility can be challenging due to proprietary datasets or models. This paper addresses the need for a more accessible, yet high-performing, neural WSD system.
Methods
The authors propose a supervised WSD approach based on a neural network architecture utilizing bidirectional Long Short-Term Memory (LSTM) cells. The core idea is to treat WSD as a classification problem where each word in a sequence is assigned a sense label.
The architecture consists of three main layers:
- Input Layer: Takes pre-trained word embeddings (e.g., GloVe) as input, representing words as dense vectors.
- Hidden Layer: Composed of bidirectional LSTM cells. These cells are adept at processing sequential data by considering both past and future context within a sequence, making them suitable for understanding word meaning in context. The "bidirectional" aspect means the network processes the sequence in both forward and backward directions, capturing richer contextual information.
- Output Layer: A fully connected layer that outputs a probability distribution over all possible senses for each input word, using a softmax function.
The training objective is to minimize the cross-entropy loss between the predicted and true sense distributions. A special <skip> token is used to ignore predictions for words that are not relevant or not annotated, improving robustness. The system is designed to be simpler and less complex to train than existing neural WSD models.
Datasets
The authors leverage several publicly available corpora for training and evaluation:
Training Corpora: * SemCor * DSO * WordNet Gloss Tagged (WNGT) * OMSTI * MASC * Ontonotes
Development Corpus: * The corpus from the SemEval 2015 Task 13 was used for development to prevent overfitting.
Evaluation Corpora: The system was evaluated on standard WSD benchmark datasets from various SemEval/SensEval campaigns:
- SensEval 2 (SE2)
- SensEval 3 (SE3)
- SemEval 2007 (SE07) tasks 07 and 17
- SemEval 2013 (SE13) task 12
- SemEval 2015 (SE15) task 13
Key Insights and Ideas
- Simplicity and Reproducibility: The paper emphasizes the importance of reproducibility by using freely accessible word embeddings and training/evaluation corpora. This contrasts with some prior state-of-the-art systems that relied on proprietary data.
- Bidirectional LSTMs for Context: The use of bidirectional LSTMs is a key architectural choice, allowing the model to capture rich contextual information from both preceding and succeeding words, which is crucial for disambiguating word senses.
- Unified Evaluation Framework: The authors utilize a unified framework for evaluating WSD systems across multiple benchmark datasets, providing a comprehensive comparison.
- Handling Unannotated Words: The
<skip>token mechanism allows the model to gracefully handle words that are not annotated in the training data, making the system more robust in real-world scenarios.
Key Results
The proposed system achieved state-of-the-art (or near state-of-the-art) F1 scores on most of the evaluated English WSD tasks.
- It achieved comparable or slightly better results than Yuan et al. (2016) on several tasks, despite using significantly less training data and publicly available resources.
- Compared to Raganato et al. (2017b), the proposed system achieved superior scores in most cases, while having a less complex architecture.
- All supervised neural network-based systems, including this one, outperformed the previous state-of-the-art non-neural system by Iacobacci et al. (2016).
- The system's performance is robust, outperforming the "most frequent sense" baseline across all evaluated tasks.
Limitations and Future Work
Limitations: * The paper acknowledges that Yuan et al. (2016) achieved slightly higher scores on some tasks, but this was attributed to their use of a proprietary language model trained on a massive corpus and a label propagation technique that is difficult to reproduce. * The DSO corpus, while used in training, is noted as being paid, though it constitutes a small percentage of the overall training data.
Future Work: The authors suggest several promising directions for future improvements:
- Implementing a label propagation step, similar to Yuan et al. (2016), to augment the training data.
- Exploring alternative fallback strategies for words where the network cannot assign a sense.
- Investigating the impact of adding attention mechanisms, which have proven effective in other NLP tasks like machine translation.
Summary updated