Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution

Nikolay Arefyev, Boris Sheludko, Alexander Podolskiy, Alexander PanchenkoCOLING 2020

TL;DR

This paper explores how to improve neural lexical substitution, a task that involves finding suitable replacement words for a target word in a given context. The authors demonstrate that by intelligently injecting information about the target word into modern language models, they can significantly boost performance on both lexical substitution and word sense induction tasks, achieving new state-of-the-art results.

What Problem Does This Paper Address?

Lexical substitution is a crucial NLP task with applications in areas like word sense disambiguation and data augmentation. The challenge lies in generating words that are not only contextually appropriate but also semantically related to the original target word. While recent advancements in large language models (LLMs) like BERT and XLNet have shown promise, they often struggle to maintain this semantic link, sometimes producing contextually fitting but semantically distant substitutes. This paper investigates how to bridge this gap by exploring various methods to inject target word information into these powerful models.

Methods

The core of this research lies in comparing different neural language models (LMs) and masked language models (MLMs) for lexical substitution and, more importantly, evaluating various "target word injection" techniques.

  • Models: The study includes established models like context2vec and ELMo, alongside more recent Transformer-based LLMs such as BERT, RoBERTa, and XLNet.
  • Target Word Injection Techniques: The authors propose and evaluate several methods to incorporate target word information:
  • +embs: This method combines the contextual probability distribution of a substitute with a distribution based on the embedding similarity between the substitute and the target word.
  • Dynamic patterns: This involves creating patterns like "target word and _" to guide the model.
  • Duplicate input: The original sentence is duplicated with the target word masked in the second instance.
  • Original input: For LLMs, the target word is left intact, allowing the model to see it directly.
  • Evaluation: Both intrinsic evaluation (on lexical substitution datasets) and extrinsic evaluation (on word sense induction datasets) are performed.

Datasets

The paper utilizes the following datasets:

  • SemEval 2007 Task 10: A benchmark for evaluating lexical substitution, focusing on polysemous words.
  • CoInCo (Concepts-In-Context): A dataset with a larger number of sentences and substitutes per example.
  • SemEval-2010 and SemEval-2013: Datasets used for extrinsic evaluation on Word Sense Induction (WSI).

Key Insights and Ideas

  • LLMs are powerful, but need guidance: While LLMs like BERT and XLNet are excellent at predicting contextually appropriate words, they benefit significantly from explicit target word information to ensure semantic relevance.
  • "+embs" is a winning strategy: The "+embs" injection method, which fuses contextual and target similarity distributions, consistently yields the best results across various models and tasks.
  • Target word injection matters: Simply masking the target word in LLMs is not enough; actively incorporating target word semantics is crucial for improving substitution quality.
  • Semantic relation analysis is insightful: Understanding the types of semantic relationships (synonymy, hypernymy, etc.) between generated substitutes and target words reveals how different models and injection methods perform.

Key Results

  • The proposed models, particularly XLNet with the "+embs" injection, achieve new state-of-the-art (SOTA) results on two Word Sense Induction datasets.
  • The "+embs" method leads to substantial improvements in Recall@10 for lexical substitution, often 2-3 times better than other injection methods for models like C2V and ELMo.
  • The XLNet+embs model outperforms previous SOTA models like PIC, c2v, and substitute vector by a large margin on intrinsic evaluation tasks.
  • The "+embs" injection method helps models preserve the correct Part-of-Speech (PoS) of the target word, reducing the number of unknown or incorrectly categorized substitutes.

Limitations and Future Work

  • Computational Cost: Some methods, like substitute validation, are computationally expensive, requiring multiple forward passes. The proposed methods aim for single forward passes.
  • Subword vs. Word-level: BERT and XLNet operate on subwords, which can limit the direct generation of multi-token substitutes compared to word-level models like ELMo.
  • Hyperparameter Tuning: The authors note that hyperparameter selection for one dataset might not be optimal for another, though they found consistent parameters in their experiments.
  • Future Directions: The paper suggests exploring individual temperature settings for each example in the "+embs" method to potentially alleviate issues where the model favors the most frequent sense over the correct one for ambiguous words. Further research into combining target word injection with other techniques like substitute validation is also a promising avenue.

Summary updated

BibTeX
@inproceedings{arefyev2020always,
  author = {Nikolay Arefyev and Boris Sheludko and Alexander Podolskiy and Alexander Panchenko},
  title = {{Always Keep your Target in Mind: Studying Semantics and Improving Performance of Neural Lexical Substitution}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2020},
}

Loading related content...

Figure