Abstract
Neural Text Simplification (TS) is primarily designed to reduce linguistic complexity for human readers. This paper explores its utility as a pre-processing tool for machine learning models in downstream Natural Language Processing (NLP) tasks. We evaluate two neural TS systems—ACCESS and NTS—across Relation Extraction (RE) and Natural Language Inference (NLI) tasks. Our results reveal a dichotomy: while simplifying inputs at inference time significantly degrades performance due to information loss, employing TS as a data augmentation technique during training consistently enhances model accuracy. We conclude that TS serves as a sophisticated semantic paraphraser, enabling models to better capture core semantic relationships across diverse linguistic structures.
Introduction
Natural Language Processing (NLP) models often struggle with the inherent "noise" of human language. Long, complex sentences with nested clauses and specialized vocabulary can obscure core meanings, leading to errors in classification. While Text Simplification (TS) has historically been used to assist children, non-native speakers, or individuals with reading disabilities, this study investigates whether TS can serve as a "pre-processor" for machines. Specifically, the authors address two questions:
- Can simplifying complex inputs at inference time make them easier for trained models to classify?
- Can TS be used as a data augmentation technique during training to provide models with more diverse, "cleaner" examples of the same semantic information?
Methodology
The researchers tested two neural TS systems: ACCESS (an aggressive simplifier) and NTS (a conservative simplifier). These systems were applied to two major NLP benchmarks:
- Relation Extraction (RE): Identifying relationships between entities (e.g., in "Steve Jobs founded Apple," the relation is org:founded_by). The study used the TACRED dataset with LSTM and SpanBERT models.
- Natural Language Inference (NLI): Determining if a "hypothesis" sentence entails, contradicts, or is neutral toward a "premise." The study used the MNLI dataset with a BERT classifier.
The study compared inference-time simplification (replacing test inputs with simplified versions) against data augmentation (adding simplified versions of training data to the original set). For Relation Extraction, a quality control step ensured that simplified sentences still contained the relevant entities.
Key Results
The experiments revealed that machines and humans "read" differently. Simplification helps humans by reducing cognitive load, but for a machine, simplifying at inference time is often destructive.
The Pitfalls of Inference-Time Simplification
Simplifying text at prediction time caused absolute performance drops of 1.38–2.80% F1 on TACRED and a staggering 11.68–49.55% drop in accuracy on MNLI. This suggests that models rely on specific linguistic features and nuances present in the original text that are stripped away during simplification.
The Benefits of Data Augmentation
In contrast, using TS for data augmentation consistently improved results across the board, regardless of whether the model used an older architecture (LSTM) or a modern transformer (SpanBERT/BERT).
Relation Extraction (TACRED)
On the TACRED dataset, models saw an improvement of 0.7–1.98% F1. As shown in Figure 4, the "Simplified + Original (AD)" configuration consistently outperformed the baseline "Original" training set for both LSTM and SpanBERT architectures.
Natural Language Inference (MNLI)
On the MNLI dataset, the BERT classifier saw an improvement of 0.50–0.65% in accuracy. Figure 5 illustrates these gains, demonstrating that augmenting the training set with a small percentage of simplified data leads to better generalization on the original test sets.
Discussion and Insights
TS augmentation works because it acts as a sophisticated form of semantic paraphrasing. It teaches the model that a core relationship (e.g., "founded by") remains valid whether expressed in a 40-word complex narrative or a 10-word simple statement.
However, information preservation is critical. The authors found that TS is most effective when "critical information"—such as the subject and object in Relation Extraction—is not lost. The benefits were also more pronounced on "harder" tasks like TACRED, which features longer sentences and fewer training examples per class than the massive MNLI dataset.
Limitations and Future Work
A major hurdle remains: Information Loss. Current neural TS models are often "black boxes" that may delete vital information or introduce hallucinations. While simple entity checks worked for Relation Extraction, defining "critical information" for tasks like NLI is more complex. Future research should focus on developing task-specific TS models that preserve the specific features required for downstream NLP tasks and creating more robust automated metrics for semantic identity.
Summary updated