TL;DR
This paper introduces PromptSub, a novel method that reframes the lexical substitution task (LST) as a causal language modeling (CLM) problem. By leveraging prompt-aware fine-tuning of decoder-only CLMs, PromptSub achieves state-of-the-art results, outperforming previous methods that relied on masked language models or encoder-decoder architectures.
What Problem Does This Paper Address?
The lexical substitution task (LST) aims to find appropriate replacements for a target word in a sentence while preserving its meaning and context. While powerful language models like GPT have shown great success in various domains, their application to LST has been limited. Existing approaches often struggle with a mismatch between the pre-training objectives of these models and the specific requirements of LST. Specifically, masked language models (MLMs), which predict words in a masked context, can lead to substitutes that alter the original sentence meaning because masking is an information-losing process. Furthermore, many LST methods rely on complex multi-stage pipelines involving external knowledge bases and post-processing, which can be cumbersome and less efficient. The authors sought to develop a more direct, end-to-end generative solution for LST using CLMs.
Methods
The core innovation of this paper is PromptSub, a system that frames LST as a causal language modeling task. Instead of masking words, PromptSub uses a prompt-aware fine-tuning strategy with decoder-only CLMs (like GPT). This approach allows the CLM to leverage the full bidirectional context of an LST instance by constructing a descriptive prompt that includes the sentence, the target word, and other relevant information. The CLM is then fine-tuned to predict specific tokens within this prompt, effectively generating substitutes.
Key aspects of their method include:
- Prompt Engineering: They developed various prompt templates, ranging from a basic one (BaseP) to more informative ones (InfoP) that include the target word's position, part-of-speech, and lemma. They also explored prompt augmentation (AugP) to incorporate multiple substitutes and external knowledge (ExP) using retrieval-augmented generation (RAG).
- Sampling Strategies: To handle the many-to-one nature of LST (one target word can have multiple valid substitutes), they introduced TopSub (selecting the top-ranked substitute) and FreqSub (sampling based on the frequency of gold substitutes in the dataset). FreqSub proved more effective.
- End-to-End Generation: PromptSub aims for a single-stage, generative solution, avoiding the need for separate generation and reranking steps or reliance on external lexical resources like WordNet.
Datasets
The authors evaluated PromptSub on several standard LST datasets:
- LS07: Used for direct comparison with previous work, particularly GeneSis.
- LS14: A combination of CoInCo, LST, and TWSI datasets, with a subset of SWORDS.
- LS21: A combination of SWORDS, LST, and TWSI datasets.
They followed established practices of merging datasets for training and evaluating on held-out portions, a common strategy due to the scarcity of LST data.
Key Insights and Ideas
- CLM for LST is Viable: The paper demonstrates that causal language models, which are inherently generative, are a powerful and effective tool for the lexical substitution task, challenging the prior focus on masked language models.
- Prompting is Key: The success of PromptSub hinges on carefully crafted prompts that provide sufficient context and guide the CLM towards generating relevant substitutes. Informative prompts (InfoP, AugP) significantly improve performance.
- Leveraging Data Frequencies: Incorporating the frequency of gold substitutes from datasets (FreqSub) is crucial for generating diverse and representative substitutions.
- Single-Stage Generation Advantage: PromptSub's end-to-end generative approach simplifies the LST pipeline and achieves competitive results without relying on external knowledge bases or complex post-processing.
- Model Capacity Matters: Scaling up the CLM (e.g., from GPT-2 to GPT-2-large) generally leads to improved performance, highlighting the potential for even better results with larger models.
Key Results
PromptSub consistently outperformed previous state-of-the-art methods across multiple datasets and metrics.
- On LS07, PromptSub achieved better results than GeneSis, even when GeneSis+WN (with WordNet post-processing) was used. For example, PromptSub achieved 21.5 in "best" and 50.7 in P@1 using gpt2-medium.
- On LS14 and LS21, PromptSub and its enhanced version PromptSub+ set new state-of-the-art results. PromptSub+ achieved an unprecedented F10 of 46.4 on LS21, outperforming prior methods by nearly 10 points.
- PromptSub demonstrated remarkable stability, showing significantly lower variance across random seeds compared to GeneSis, attributed to its greedy sampling from a single-step probability distribution.
- The paper also showed that larger models (GPT-2-large) generally performed better than smaller ones (GPT-2-medium, GPT-2), indicating the benefit of increased model capacity.
Limitations and Future Work
The authors acknowledge several limitations:
- Data Scarcity: The effectiveness of PromptSub, like other supervised methods, is limited by the quality and diversity of available training data. Scaling up data is a key challenge.
- Computational Demands: Fine-tuning large CLMs can be computationally expensive, potentially limiting practicality in resource-constrained environments.
- Latest PLMs: PromptSub has not been tested with the very latest, often proprietary, PLMs due to limited access and computational resources.
Future work suggested includes:
- Exploring Larger Models and More Data: Investigating the performance of PromptSub with more powerful models (like GPT-3 and beyond) and larger LST datasets.
- Advanced Prompt Engineering: Further exploring prompt design and retrieval-augmented generation (RAG) to incorporate external knowledge more effectively.
- Broader Applicability: Demonstrating the utility of CLMs for other semantic tasks beyond LST.
Summary updated