Abstract
Lexical substitution involves identifying appropriate replacements for a target word in a given context while preserving the original meaning. While Transformer-based models like BERT excel at contextual understanding, they often lack the precise semantic grounding found in structured lexical resources. We present LexSubCon, a framework that bridges this gap by integrating WordNet knowledge into contextual embeddings. By employing a novel "mix-up" embedding strategy and leveraging dictionary definitions (glosses), LexSubCon achieves state-of-the-art performance on standard benchmarks, outperforming previous methods by significant margins.
Introduction
The task of lexical substitution is fundamental to many natural language processing applications, including text simplification and data augmentation. Modern language models rely heavily on distributional semantics, which can sometimes lead to contextually plausible but semantically incorrect suggestions. For example, a model might suggest "positive" as a substitute for "terrible" in a context where both words frequently appear, despite their opposite meanings. LexSubCon addresses this by anchoring contextual embeddings with structured knowledge from external lexical databases like WordNet.
Methodology
The LexSubCon framework consists of three primary components designed to refine candidate extraction and scoring.
1. Mix-Up Embedding Strategy
Unlike traditional masking or dropout techniques, LexSubCon utilizes a "mix-up" strategy during candidate extraction. The target word's input embedding is linearly interpolated with the average embedding of its WordNet synonyms. This approach provides the model with a semantic "hint," guiding it toward the correct neighborhood of meaning without completely obscuring the original word.
2. Gloss-Sentence Similarity
To further refine candidates, the model incorporates dictionary definitions (glosses). A Word Sense Disambiguation (WSD) model identifies the correct sense of the target word. LexSubCon then calculates the cosine similarity between the embeddings of the target word's definition and the candidate word's definition, ensuring semantic alignment at the definitional level.
3. Fine-tuned Sentence Similarity
Finally, a RoBERTa-based model evaluates the similarity between the original sentence and the sentence containing the substitute. To improve robustness, the authors used back-translation to generate synthetic training data, ensuring the model can distinguish between meaning-preserving and meaning-altering substitutions.
Experimental Results
LexSubCon was evaluated on two major benchmarks: the SemEval 2007 (LS07) dataset and the Concepts-In-Context (CoInCo) dataset.
As shown in the results table, LexSubCon achieved a new state-of-the-art across all metrics. On the LS07 "best" metric, it reached a score of 21.1, nearly doubling the performance of previous BERT-based approaches (12.8). Similar gains were observed on the larger CoInCo dataset.
Key Insights
- Semantic Anchoring: External resources act as a necessary constraint on the flexibility of contextual models, preventing semantic drift.
- Superiority of Mix-up: Providing partial information via synonym interpolation is more effective for lexical substitution than the standard
[MASK]token approach. - Definition-level Features: Glosses provide high-fidelity signals for synonymy that are often absent in word-level contextual embeddings.
Conclusion and Future Work
LexSubCon demonstrates that integrating structured lexical knowledge significantly enhances the performance of contextual models in lexical substitution tasks. While currently focused on English and dependent on WordNet, future iterations could expand to multilingual resources like BabelNet and explore optimizations to reduce the computational overhead of definition-based scoring.
Summary updated