LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution

George Michalopoulos, Ian McKillop, Alexander Wong, Helen H. Chen

Abstract

Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy in the creation of the input embedding of the target word through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution in the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.

Introduction

Lexical substitution involves identifying words that can replace a target word in a sentence without altering its original meaning. While modern contextual word embedding models like BERT have significantly advanced this field by leveraging surrounding context, they often operate in a vacuum, ignoring structured human knowledge found in lexical databases like WordNet. This reliance on context alone can lead to errors where models suggest words that fit grammatically but fail semantically, such as confusing antonyms. LexSubCon addresses this by integrating external lexical resources directly into the embedding and scoring process.

Methodology

LexSubCon is an end-to-end framework that utilizes four distinct scoring mechanisms to rank potential substitutes.

  1. Mix-Up Embedding Strategy: Rather than masking the target word or using simple dropout, LexSubCon creates a synthetic embedding. It linearly interpolates the target word's embedding with the average embedding of its known synonyms from WordNet, providing a semantic "anchor" that guides the model toward the correct meaning.
  2. Gloss-Sentence Similarity: The model incorporates dictionary definitions (glosses). By using a Word Sense Disambiguation (WSD) model to select the contextually appropriate definition, it calculates the semantic similarity between the glosses of the target word and its candidates.
  3. Sentence Similarity Score: To ensure the overall sentence meaning remains intact, a fine-tuned RoBERTa model compares the original sentence with the modified version. This component was trained using back-translation to ensure robustness.
  4. Candidate Validation: This metric measures the influence of the substitution on the sentence's semantics by analyzing self-attention patterns within the model.
The LexSubCon framework architecture, illustrating the Candidate Extraction and Scoring Modules.

Evaluation and Results

The framework was evaluated on two gold-standard English datasets: SemEval 2007 (LS07) and Concepts-In-Context (CoInCo). LexSubCon achieved new state-of-the-art (SOTA) results across all official metrics, including Precision@1 and Generalized Average Precision (GAP).

Performance comparison of LexSubCon against baseline and state-of-the-art methods on LS07 and CoInCo datasets.

As shown in the results, LexSubCon significantly outperforms previous methods. On the LS07 dataset, it achieved a Precision@1 of 51.7%, a substantial jump from the previous SOTA of 31.7%. Ablation studies confirmed that each component—Gloss similarity, Mix-up strategy, and Sentence similarity—contributes significantly to the overall performance. Furthermore, LexSubCon proved effective for data augmentation, outperforming standard techniques like Easy Data Augmentation (EDA) when generating synthetic training data for text classification.

Conclusion

LexSubCon demonstrates that the integration of structured lexical knowledge with contextual embeddings is a powerful approach for lexical substitution. By bridging the gap between deep learning models and traditional lexical resources, the framework avoids common semantic pitfalls and sets a new benchmark for the task. Future work may involve extending this framework to low-resource languages and exploring more computationally efficient architectures to reduce the overhead of multi-component scoring.

Summary updated

Abstract

Lexical substitution is the task of generating meaningful substitutes for a word in a given textual context. Contextual word embedding models have achieved state-of-the-art results in the lexical substitution task by relying on contextual information extracted from the replaced word within the sentence. However, such models do not take into account structured knowledge that exists in external lexical databases. We introduce LexSubCon, an end-to-end lexical substitution framework based on contextual embedding models that can identify highly accurate substitute candidates. This is achieved by combining contextual information with knowledge from structured lexical resources. Our approach involves: (i) introducing a novel mix-up embedding strategy in the creation of the input embedding of the target word through linearly interpolating the pair of the target input embedding and the average embedding of its probable synonyms; (ii) considering the similarity of the sentence-definition embeddings of the target word and its proposed candidates; and, (iii) calculating the effect of each substitution in the semantics of the sentence through a fine-tuned sentence similarity model. Our experiments show that LexSubCon outperforms previous state-of-the-art methods on LS07 and CoInCo benchmark datasets that are widely used for lexical substitution tasks.
BibTeX
@inproceedings{michalopoulos2022lexsubcon,
  author = {George Michalopoulos and Ian McKillop and Alexander Wong and Helen H. Chen},
  title = {{LexSubCon: Integrating Knowledge from Lexical Resources into Contextual Embeddings for Lexical Substitution}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=Lohbf7WxmA},
}
Figure