A Deep Dive into Word Sense Disambiguation with LSTM

Minh Le, Marten Postma, Jacopo Urbani, Piek VossenCOLING 2018

TL;DR

This paper reproduces and analyzes a state-of-the-art LSTM-based Word Sense Disambiguation (WSD) method. They demonstrate that similar performance can be achieved with significantly less training data than previously thought, and provide insights into the model's strengths and weaknesses, particularly concerning sense coverage in annotated datasets.

What Problem Does This Paper Address?

Word Sense Disambiguation (WSD) is a fundamental task in Natural Language Processing (NLP) that aims to identify the correct meaning of a word in a given context, drawing from a lexical database like WordNet. While deep learning models, particularly LSTMs, have shown great promise in WSD, a highly effective LSTM-based approach by Yuan et al. (2016) had its training data and code withheld. This lack of reproducibility hinders further research, making it difficult to understand the method's limitations and potential for improvement. This paper addresses this by reproducing Yuan et al.'s method, making their code and models available, and conducting a thorough analysis of its performance and underlying factors.

Methods

The core of the paper is the reproduction of Yuan et al.'s (2016) LSTM-based WSD method. This involves:

  1. Constructing Language Models: An LSTM network is trained on a large unannotated corpus to capture word meanings in context. This model generates "context vectors" for sentences.
  2. Calculating Sense Embeddings: Using a smaller, sense-annotated dataset, the LSTM model is used to compute context vectors for each occurrence of a specific word sense. The "sense embedding" for that sense is then the average of these context vectors.
  3. Predicting Senses: For a given target word in an unseen sentence, the model first generates a context vector. Then, it compares this context vector to the pre-computed sense embeddings for the target word's possible senses (retrieved from WordNet). The sense with the closest embedding (using cosine similarity) is chosen.
  4. Label Propagation (Optional): As an alternative to simple averaging, the authors also implement a label propagation technique. This method leverages both annotated and unannotated data to propagate sense labels, aiming to capture more complex relationships and potentially reduce bias towards the most frequent sense.

Datasets

  • Training Data (Language Model):
    • Gigaword Fifth Edition: A corpus of 1.8 billion tokens. This is significantly smaller than the 100 billion tokens used by Yuan et al. (2016).
  • Training Data (Sense Embeddings):
    • SemCor: Approximately 240,000 sense-annotated words.
  • OMSTI: One million automatically tagged sense annotations.
  • Evaluation Datasets:
    • Senseval-2 (All-Words Task): A dataset for evaluating WSD on nouns, verbs, adjectives, and adverbs.
  • SemEval-2013 (Task 12): A dataset focusing on noun disambiguation.

Key Insights and Ideas

  • Data Efficiency: State-of-the-art WSD performance can be achieved with substantially less unannotated data (1.8 billion tokens vs. 100 billion tokens) than previously suggested.
  • Diminishing Returns: While more unannotated data generally improves performance, the gains become exponentially smaller as the corpus size increases.
  • Balanced Sense Assignment: The LSTM model shows a less-pronounced bias towards the most frequent sense (MFS) compared to some other methods, performing relatively well on less-frequent senses.
  • Sense Coverage is Crucial: The limited coverage of senses in annotated datasets is a major bottleneck. If a word's correct sense is not represented in the training data, the model cannot learn to identify it, even with a powerful LSTM. This limits the achievable performance.
  • Sensekey vs. Synset: Disambiguating at the "sensekey" level (a more granular representation than WordNet synsets) leads to better performance, suggesting that finer-grained sense distinctions are more discriminative.

Key Results

  • Reproduced Performance: The reproduced LSTM model achieved competitive results, coming close to state-of-the-art performance on both Senseval-2 and SemEval-2013 datasets, despite using significantly less training data.
  • Senseval-2: Achieved an F₁ score of 0.720 using SemCor for training sense embeddings.
  • SemEval-2013: Achieved an F₁ score of 0.656 using SemCor+OMSTI for training sense embeddings.
  • MFS Fallback Importance: The MFS fallback strategy (defaulting to the most frequent sense when no other information is available) plays a significant role in achieving high overall accuracy, especially when sense coverage is low. Without it, performance can drop considerably.
  • Label Propagation vs. Averaging: While label propagation was competitive, it did not consistently outperform the simpler averaging technique in their experiments, particularly when sense coverage was low.

Limitations and Future Work

  • Sense Coverage: The primary limitation identified is the insufficient coverage of senses in existing annotated datasets. This prevents the model from learning to disambiguate all possible meanings, capping overall performance. Future work could focus on creating larger, more comprehensive annotated datasets.
  • Data Size vs. Model Capacity: The paper suggests that while increasing unannotated data and model size can improve performance, these gains are subject to diminishing returns and require exponentially more resources.
  • Sensekey Granularity: The authors suggest that exploring even finer-grained sense representations beyond sensekeys could be a direction for future research.
  • Reproducibility: The authors have made their code and trained models publicly available, facilitating further research and validation.

Summary updated

BibTeX
@inproceedings{le2018deep,
  author = {Minh Le and Marten Postma and Jacopo Urbani and Piek Vossen},
  title = {{A Deep Dive into Word Sense Disambiguation with LSTM}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2018},
}
Figure