Supervised All-Words Lexical Substitution using Delexicalized Features

György Szarvas, Chris Biemann, Iryna GurevychNAACL 2013 · OpenReview

Abstract

We propose a supervised lexical substitution system that does not use separate classifiers per word and is therefore applicable to any word in the vocabulary. Instead of learning word-specific substitution patterns, a global model for lexical substitution is trained on delexicalized (i.e., non-lexical) features, which allows us to exploit the power of supervised methods while being able to generalize beyond target words in the training set. This way, our approach remains technically straightforward, provides better performance, and offers similar coverage in comparison to unsupervised approaches. Using features from lexical resources, as well as a variety of features computed from large corpora (n-gram counts, distributional similarity) and a ranking method based on the posterior probabilities obtained from a Maximum Entropy classifier, we improve over the state of the art in the LexSub Best-Precision metric and the Generalized Average Precision measure. Robustness of our approach is demonstrated by evaluating it successfully on two different datasets.

Introduction

Lexical substitution—the task of identifying contextually appropriate synonyms for a target word—is a cornerstone of natural language processing with applications in text simplification, plagiarism detection, and information retrieval. Historically, this task has been divided between unsupervised methods, which offer broad coverage but lower accuracy, and supervised "lexical sample" methods, which achieve high accuracy but require thousands of training examples for every specific word.

This paper introduces a supervised "all-words" system that bridges this gap. By utilizing delexicalized features, we develop a single global model capable of generalizing to any word in the vocabulary, including those not seen during training.

Methodology: Delexicalized Features

The core innovation lies in treating lexical substitution as a ranking problem solved by a Maximum Entropy (MaxEnt) classifier. Instead of learning patterns tied to specific lexical items (e.g., "if the word is bank..."), the model learns from abstract, non-lexical indicators that describe the relationship between a candidate substitute and its context.

The feature set includes:

  • Local n-gram counts: Frequency of the candidate in 3-word and 5-word windows within a massive web corpus.
  • Lexical Resource features: Direct synonyms and sense rankings from WordNet (WN).
  • Distributional Similarity: Overlap in "neighbor" words between the target and candidate in large corpora.
  • Shallow Syntactic features: Part-of-Speech (POS) tags of surrounding words.

Feature Contribution Analysis

To understand the impact of these features, an ablation study was conducted. While local n-grams provide the strongest individual signal, the combination of all features yields the best performance, demonstrating a significant synergy between local contextual "sound" and global semantic relations.

Ablation study showing the impact of removing specific feature sets on GAP and P@1 metrics.

Experimental Evaluation

The system was evaluated on two distinct datasets to ensure robustness across domains:

  1. LexSub (SemEval 2007): A standard benchmark with 201 target words and 2,002 sentences.
  2. TWSI (Turk Bootstrap Word Sense Inventory): A larger-scale dataset with over 1,000 nouns and nearly 25,000 sentences.

Results and Discussion

The delexicalized global model significantly outperformed previous state-of-the-art (SOTA) systems and competitive baselines. On the LexSub dataset, the model achieved a Best-Precision of 15.94 and a Generalized Average Precision (GAP) of 52.4, setting a new benchmark for the task.

Comparison of the proposed model against existing state-of-the-art systems on the LexSub benchmark.

The results on the TWSI dataset further confirmed the system's robustness, showing consistent improvements even as the scale of the data increased. This suggests that the "physics" of lexical substitution—how words fit into contexts—can indeed be learned as a general linguistic property rather than a collection of word-specific rules.

Conclusion

By shifting from word-specific classifiers to a global model based on delexicalized features, this research provides a scalable and high-performing solution for all-words lexical substitution. Future work may address the limitations of relying on fixed lexical resources like WordNet by exploring automated candidate generation from corpora and investigating more complex "learning to rank" architectures.

Summary updated

Abstract

We propose a supervised lexical substitution system that does not use separate classifiers per word and is therefore applicable to any word in the vocabulary. Instead of learning word-specific substitution patterns, a global model for lexical substitution is trained on delexicalized (i.e., non lexical) features, which allows to exploit the power of supervised methods while being able to generalize beyond target words in the training set. This way, our approach remains technically straightforward, provides better performance and similar coverage in comparison to unsupervised approaches. Using features from lexical resources, as well as a variety of features computed from large corpora (n-gram counts, distributional similarity) and a ranking method based on the posterior probabilities obtained from a Maximum Entropy classifier, we improve over the state of the art in the LexSub Best-Precision metric and the Generalized Average Precision measure. Robustness of our approach is demonstrated by evaluating it successfully on two different datasets.
BibTeX
@inproceedings{szarvas2013supervised,
  author = {György Szarvas and Chris Biemann and Iryna Gurevych},
  title = {{Supervised All-Words Lexical Substitution using Delexicalized Features}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2013},
  url = {https://openreview.net/forum?id=ryWP5QZuZB},
}
Figure