Introduction
Smart Word Substitution (SWS) aims to enhance text quality by suggesting contextually appropriate word alternatives. Unlike traditional lexical substitution, which focuses on semantic equivalence, SWS seeks to improve the "naturalness" or impact of a sentence. However, developing robust SWS systems is hindered by the inherent subjectivity of word choice. Human annotators often disagree on optimal substitutions, making ground-truth labels sparse and expensive to obtain. This paper proposes a novel framework that bypasses human labeling by using automated quality metrics as a proxy for training.
The Challenge of Subjectivity
A core motivation for this work is the observation that word substitution is rarely a binary "right or wrong" task. Analysis of the SWS benchmark reveals that in 94% of cases, both humans and models prefer to leave a token unchanged. When a change is suggested, the agreement between different annotators or between humans and models is remarkably low (approximately 11%).
This high level of disagreement suggests that training models to predict a single "correct" human label is suboptimal. Instead, the task is better framed as learning the relative quality of various candidates.
Methodology: Ranking with Model-based Scores
The researchers introduce a self-supervised approach that treats word substitution as a ranking problem guided by BARTScore, a metric that estimates sentence quality based on the log-likelihood of a pre-trained BART model.
1. BARTScore as a Teacher
BARTScore serves as the objective function. By calculating the score for a sentence before and after a substitution, the system can determine if a specific word choice improves grammatical correctness or semantic flow.
2. Margin Ranking (MR) Loss
Rather than performing classification, the model is trained using a Margin Ranking loss. For a set of $K$ candidate substitutes, the model learns to assign higher probabilities to words that result in higher BARTScores. This ensures the model understands the relative hierarchy of word quality.
3. Average Score (AS) Improvement
To prevent the model from simply ranking mediocre options, an Average Score (AS) improvement term ($L_{AS}$) is added. This term encourages the model to maximize the weighted average quality score across its top-ranked predictions, pushing the model toward high-quality suggestions.
The final model, MR+AS, is built upon a fine-tuned BERT-base-uncased architecture.
Experimental Results
The MR+AS model was evaluated against several baselines, including vanilla masked language models (BERT, BART) and state-of-the-art Large Language Models (GPT-4o, LLaMA-3.1), across four datasets: SWS, LS07, LS14, and XSum.
Comparative Performance
The primary metrics for evaluation were Cosine Similarity (CS), measuring the alignment between model likelihoods and BARTScores, and the Average BARTScore Ratio (ABR), measuring the quality of the top suggestions.
As shown in the results, MR+AS achieved a median CS of 0.99, significantly outperforming vanilla BERT (0.92) and even LLaMA (0.94). In terms of quality (ABR), the specialized MR+AS model (0.86) matched or exceeded the performance of GPT-4o (0.86) across the board. This indicates that for high-precision tasks like word substitution, a well-tuned smaller model can outperform much larger general-purpose LLMs.
Human Validation
A human preference study further validated these findings. When the MR+AS model and BARTScore agreed on a ranking, human annotators favored the suggested choice 79% of the time. Even in cases of disagreement, humans sided with the BARTScore-preferred choice 67% of the time, confirming that model-based scores are effective proxies for human judgment.
Conclusion
This research demonstrates that model-based quality metrics like BARTScore have matured sufficiently to serve as "label proxies" for training specialized language models. By framing SWS as a ranking problem and optimizing for relative quality rather than absolute labels, the MR+AS framework achieves state-of-the-art performance while eliminating the need for costly human annotation. Future work will explore the integration of other scoring metrics and methods to increase the substitution rate without sacrificing quality.
Summary updated