Abstract
Smart word substitution aims to enhance sentence quality by improving word choices; however current benchmarks rely on human-labeled data. Since word choices are inherently subjective, ground-truth word substitutions generated by a small group of annotators are often incomplete and likely not generalizable. To circumvent this issue, we instead employ a model-based score (BARTScore) to quantify sentence quality, thus forgoing the need for human annotations. Specifically, we use this score to define a distribution for each word substitution, allowing one to test whether a substitution is statistically superior relative to others. In addition, we propose a loss function that directly optimizes the alignment between model predictions and sentence scores, while also enhancing the overall quality score of a substitution. Crucially, model learning no longer requires human labels, thus avoiding the cost of annotation while maintaining the quality of the text modified with substitutions. Experimental results show that the proposed approach outperforms both masked language models (BERT, BART) and large language models (GPT-4, LLaMA).
Introduction
Smart Word Substitution (SWS) is the task of identifying words in a sentence that could be improved and suggesting better alternatives. This task is critical for applications like writing assistants and automated editing. Traditionally, training SWS models has required massive datasets labeled by humans. However, word choice is inherently subjective; a small group of annotators might only suggest a few synonyms for a word, even if many valid alternatives exist. This makes human-labeled "ground truth" incomplete and expensive to produce. Furthermore, standard language models often prioritize fluency over quality, failing to identify words that actually improve the sentence's impact.
Methodology
To address the limitations of human labeling, we propose a model-based approach that replaces human judgment with an automated scoring metric.
BARTScore as a Quality Proxy
Instead of relying on human labels, we use BARTScore, a metric that calculates the log-likelihood of a sentence using a pre-trained BART model. High BARTScores correlate strongly with human perceptions of text quality, providing a reliable signal for training.
Statistical Hypothesis Testing
To determine if a substitution is genuinely superior, we calculate a p-value by comparing a candidate word against a reference distribution of 1,000 other potential substitutes. This ensures that any suggested improvement is statistically significant rather than a marginal fluctuation in the scoring metric.
Preference-Aware Learning (MR+AS)
We introduce a novel training objective called MR+AS (Margin Ranking + Average Score):
- Margin Ranking (MR): This component forces the model to rank word candidates in the same order as their BARTScores, ensuring the model learns relative quality.
- Average Score (AS): This component directly optimizes the model to increase the overall quality score of the resulting sentence.
Experimental Results
We evaluated our approach across four diverse datasets: SWS (Wikipedia-based), LS07 and LS14 (Lexical Substitution benchmarks), and XSum (summarization data).
Performance Comparison
Our proposed MR+AS model achieved a median Cosine Similarity (CS) of 0.98–0.99 across all datasets, indicating near-perfect alignment with quality scores. As shown in the performance table below, MR+AS significantly outperforms state-of-the-art large language models like GPT-4o and LLaMA, as well as standard BERT and BART baselines.
The Incompleteness of Human Annotations
A key finding of our research is the "incompleteness" of human-labeled data. In 74% of cases where a human annotator suggested a change, our model chose a different word. Crucially, these model-driven changes often resulted in higher BARTScores than the human suggestions, proving that automated metrics can identify high-quality vocabulary choices that human annotators frequently overlook.
Conclusion
By leveraging model-based scoring and a ranking-based loss function, we have demonstrated that it is possible to train high-performing word substitution models without human labels. Our MR+AS approach not only avoids the high cost of manual annotation but also identifies superior word choices that humans miss, outperforming even the most advanced large language models in alignment with text quality metrics.
Summary updated