ALaSca: an Automated approach for Large-Scale Lexical Substitution

Caterina Lacerra, Tommaso Pasini, Rocco Tripodi, Roberto NavigliIJCAI 2021 · DOI

TL;DR

This paper introduces ALaSca, a novel automated approach for generating large-scale datasets for lexical substitution. By leveraging pre-trained language models and Wikipedia, ALaSca creates diverse and high-quality training data, enabling transformer-based models to achieve state-of-the-art performance on this task, even outperforming models trained on manually curated datasets.

What Problem Does This Paper Address?

The lexical substitution task aims to find suitable replacements for a target word within a given context, ensuring the overall meaning of the sentence remains unchanged. This task is valuable for applications like word sense induction and text simplification. However, a significant bottleneck has been the scarcity of large, high-quality annotated datasets. Existing approaches often rely on unsupervised methods due to this data limitation, which restricts the full potential of powerful pre-trained neural language models like BERT and GPT. Manually creating such datasets is time-consuming and expensive, hindering progress in the field.

Methods

ALaSca tackles the data scarcity problem by proposing an automated pipeline for dataset generation. The core steps are:

  1. Sentence Retrieval and Clustering: ALaSca starts by retrieving sentences from a corpus (like Wikipedia) that contain the target word. To ensure diversity of meanings, these sentences are then clustered based on contextualized embeddings of the target word. This grouping allows for the sampling of sentences representing different senses of the word.
  2. Substitutes Selection: For each sampled sentence, ALaSca identifies similar sentences from the corpus where the target word does not appear. By analyzing the words in these similar sentences, potential substitutes for the target word are extracted.
  3. Substitutes Ranking and Dataset Construction: The extracted substitutes are then ranked based on their contextual similarity to the original target word and its context. Finally, the sampled sentences are paired with their ranked lists of substitutes to form the training dataset.

The paper explores different strategies for clustering (fixed vs. adaptive number of clusters) and sentence embedding models (LASER, Sentence-BERT), as well as similarity measures (contextualized vs. combined similarity) to optimize the dataset generation process.

Datasets

  • Training Data Generation: ALaSca generates its own "silver" datasets. The paper describes generating a dataset named ALaScar using Wikipedia as the source corpus.
  • Evaluation Datasets:
    • LST (Lexical Substitution Task) dataset: The original dataset for the task, containing 2010 sentences with one annotated word per sentence.
  • TWSI (Turk bootstrap Word Sense Inventory): A larger dataset with around 25K Wikipedia sentences, focusing on nouns.
  • CoInCo (Concept In Context): A dataset derived from the MASC corpus, aiming to cover diverse parts of speech.
  • Development Set: A combination of the development split of CoInCo and 30% of TWSI instances.

Key Insights and Ideas

  • Automated Data Generation is Viable: The most significant insight is that high-quality lexical substitution datasets can be automatically generated, significantly reducing the reliance on manual annotation.
  • Clustering for Sense Diversity: Grouping sentences based on contextualized embeddings is crucial for capturing the different meanings (senses) of a target word, leading to more comprehensive datasets.
  • Combined Similarity is Key: Using a combination of contextualized and static word embedding similarities for ranking substitutes generally yields better results than using either alone.
  • Finetuning is Highly Beneficial: The paper demonstrates that finetuning a BERT model on lexical substitution data, even automatically generated data, leads to substantial performance gains compared to unsupervised approaches.
  • ALaSca Data Enables State-of-the-Art: A simple BERT model trained on ALaSca data can achieve performance comparable to or exceeding complex state-of-the-art models trained on manually curated datasets.

Key Results

  • A BERT model finetuned on ALaSca data achieved a GAP score of 58.2 on the LST test set, which is competitive with, and in some cases surpasses, models trained on manually curated datasets like CoInCo and TWSI.
  • When ALaSca data was concatenated with gold standard datasets, the model reached 60.5, matching the performance of the best reported models.
  • The paper shows that ALaSca data allows a simple BERT model to achieve 6 points higher than comparable unsupervised approaches.
  • In a zero-shot setting (where test targets are removed from training), ALaSca data enabled better generalization than using only gold training data.

Limitations and Future Work

  • BPE Tokenization Limitation: The paper acknowledges that BPE tokenization inherently limits the set of possible substitutes to those not split into multiple subwords, potentially missing up to 40% of candidate substitutes for 50% of LST test instances.
  • Sub-optimal Clusterings: While adaptive clustering was explored, the fixed clustering approach (based on WordNet senses) generally performed better, suggesting that manual resources can still inform automated processes.
  • Future Work: The authors suggest exploring ALaSca's application to other languages and its use in downstream applications like text simplification. They also plan to investigate ways to improve the handling of subword tokenization limitations.

Summary updated

BibTeX
@inproceedings{lacerra2021alasca,
  author = {Caterina Lacerra and Tommaso Pasini and Rocco Tripodi and Roberto Navigli},
  title = {{ALaSca: an Automated approach for Large-Scale Lexical Substitution}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.24963/ijcai.2021/528},
}
Figure