Abstract
Pre-trained Language Models (PrLMs) such as BERT and ELECTRA have significantly advanced natural language processing but remain sensitive to lexical diversity, which often acts as semantic noise. This paper introduces LRLS (Lemmatization and Rare word replacement), a lightweight, rule-based lexical simplification method designed to enhance model robustness. By integrating simplified text as an auxiliary input (LRLS-Aux), the proposed framework achieves consistent performance gains across five standard text classification benchmarks while maintaining high computational efficiency.
Introduction
Despite their success, PrLMs are surprisingly fragile; minor lexical variations, such as synonym substitutions or tense changes, can lead to erroneous predictions. This suggests that the vast variety of ways to express the same concept—lexical diversity—can hinder a model's ability to capture core semantic meanings. While Lexical Simplification (LS) techniques exist, they are typically optimized for human readability and are computationally intensive. This work proposes a fast, model-centric simplification process that can be seamlessly integrated into existing machine learning pipelines.
Methodology
The proposed approach consists of two primary components: the LRLS simplification process and the LRLS-Aux auxiliary input framework.
LRLS: Lemmatization and Rare Word Replacement
LRLS is a rule-based method that avoids the overhead of neural LS models:
- Lemmatization: Utilizing the Natural Language Toolkit (NLTK), verbs are converted to infinitive forms and nouns to singular forms, stripping away grammatical noise.
- Rare Word Replacement: Words falling below a frequency threshold ($n_f$) are identified as "rare." These are replaced with common synonyms retrieved from specialized word embeddings, ensuring the replacement shares the same Part-of-Speech (POS).
LRLS-Aux Framework
Rather than replacing the original input, the simplified sentence is used as an auxiliary input. The original and simplified versions are concatenated using a separator token ([SEP]), allowing the model to leverage both the nuanced original text and the clarified simplified version during training and inference.
Experimental Results
The LRLS-Aux framework was evaluated on five benchmarks: SST-2, CR, SUBJ, MR, and AG. The method demonstrated consistent improvements over strong baselines.
As shown in Table 2, LRLS-Aux improved BERT-base performance by an average of 1.1% and ELECTRA-large by 0.7%. Notably, the method provided statistically significant boosts even for high-performing models like ELECTRA-large, indicating that even state-of-the-art models struggle with lexical noise.
Discussion and Key Insights
- Efficiency: LRLS is over 100 times faster than neural-based methods like BERT-LS, as it avoids recursive neural network passes.
- Dual-Perspective Learning: Concatenating the original and simplified text is superior to using simplified text alone. The simplified version acts as a "cleaner" view that helps the model verify its decisions on the original text.
- Inference-Time Benefits: Unlike standard data augmentation, using simplification as an auxiliary input during inference provides a more substantial performance boost.
- Parameter Sensitivity: The researchers identified a "Goldilocks" zone for simplification; excessive replacement can alter the sentence's original meaning, while insufficient simplification yields no gain.
Conclusion
LRLS-Aux offers a computationally efficient way to improve the robustness and accuracy of PrLMs by mitigating the impact of lexical noise. Future work will explore the application of this auxiliary framework to more complex tasks such as Question Answering and Natural Language Inference, as well as more sophisticated integration methods for simplified text.
Summary updated