LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

George Michalopoulos, Ian McKillop, Alexander Wong, Helen H. Chen

Abstract

Lexical substitution involves identifying appropriate replacements for a target word in a given context while preserving the original meaning. While Transformer-based models like BERT excel at contextual understanding, they often lack the precise semantic grounding found in structured lexical resources. We present LexSubCon, a framework that bridges this gap by integrating WordNet knowledge into contextual embeddings. By employing a novel "mix-up" embedding strategy and leveraging dictionary definitions (glosses), LexSubCon achieves state-of-the-art performance on standard benchmarks, outperforming previous methods by significant margins.

Introduction

The task of lexical substitution is fundamental to many natural language processing applications, including text simplification and data augmentation. Modern language models rely heavily on distributional semantics, which can sometimes lead to contextually plausible but semantically incorrect suggestions. For example, a model might suggest "positive" as a substitute for "terrible" in a context where both words frequently appear, despite their opposite meanings. LexSubCon addresses this by anchoring contextual embeddings with structured knowledge from external lexical databases like WordNet.

Methodology

The LexSubCon framework consists of three primary components designed to refine candidate extraction and scoring.

The LexSubCon architecture, illustrating the Candidate Extraction and Scoring Modules.

1. Mix-Up Embedding Strategy

Unlike traditional masking or dropout techniques, LexSubCon utilizes a "mix-up" strategy during candidate extraction. The target word's input embedding is linearly interpolated with the average embedding of its WordNet synonyms. This approach provides the model with a semantic "hint," guiding it toward the correct neighborhood of meaning without completely obscuring the original word.

2. Gloss-Sentence Similarity

To further refine candidates, the model incorporates dictionary definitions (glosses). A Word Sense Disambiguation (WSD) model identifies the correct sense of the target word. LexSubCon then calculates the cosine similarity between the embeddings of the target word's definition and the candidate word's definition, ensuring semantic alignment at the definitional level.

3. Fine-tuned Sentence Similarity

Finally, a RoBERTa-based model evaluates the similarity between the original sentence and the sentence containing the substitute. To improve robustness, the authors used back-translation to generate synthetic training data, ensuring the model can distinguish between meaning-preserving and meaning-altering substitutions.

Experimental Results

LexSubCon was evaluated on two major benchmarks: the SemEval 2007 (LS07) dataset and the Concepts-In-Context (CoInCo) dataset.

Comparison of LexSubCon against previous state-of-the-art methods on LS07 and CoInCo datasets.

As shown in the results table, LexSubCon achieved a new state-of-the-art across all metrics. On the LS07 "best" metric, it reached a score of 21.1, nearly doubling the performance of previous BERT-based approaches (12.8). Similar gains were observed on the larger CoInCo dataset.

Key Insights

  • Semantic Anchoring: External resources act as a necessary constraint on the flexibility of contextual models, preventing semantic drift.
  • Superiority of Mix-up: Providing partial information via synonym interpolation is more effective for lexical substitution than the standard [MASK] token approach.
  • Definition-level Features: Glosses provide high-fidelity signals for synonymy that are often absent in word-level contextual embeddings.

Conclusion and Future Work

LexSubCon demonstrates that integrating structured lexical knowledge significantly enhances the performance of contextual models in lexical substitution tasks. While currently focused on English and dependent on WordNet, future iterations could expand to multilingual resources like BabelNet and explore optimizations to reduce the computational overhead of definition-based scoring.

Summary updated

Abstract

Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy in the creation of the input embedding of the target word through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution in the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.
BibTeX
@inproceedings{michalopoulos2022lexsubcon,
  author = {George Michalopoulos and Ian McKillop and Alexander Wong and Helen H. Chen},
  title = {{LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Lohbf7WxmA},
}
Figure