How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks

Hoang Van, Zheng Tang, Mihai Surdeanu

Abstract

Neural Text Simplification (TS) is primarily designed to reduce linguistic complexity for human readers. This paper explores its utility as a pre-processing tool for machine learning models in downstream Natural Language Processing (NLP) tasks. We evaluate two neural TS systems—ACCESS and NTS—across Relation Extraction (RE) and Natural Language Inference (NLI) tasks. Our results reveal a dichotomy: while simplifying inputs at inference time significantly degrades performance due to information loss, employing TS as a data augmentation technique during training consistently enhances model accuracy. We conclude that TS serves as a sophisticated semantic paraphraser, enabling models to better capture core semantic relationships across diverse linguistic structures.

Introduction

Natural Language Processing (NLP) models often struggle with the inherent "noise" of human language. Long, complex sentences with nested clauses and specialized vocabulary can obscure core meanings, leading to errors in classification. While Text Simplification (TS) has historically been used to assist children, non-native speakers, or individuals with reading disabilities, this study investigates whether TS can serve as a "pre-processor" for machines. Specifically, the authors address two questions:

  1. Can simplifying complex inputs at inference time make them easier for trained models to classify?
  2. Can TS be used as a data augmentation technique during training to provide models with more diverse, "cleaner" examples of the same semantic information?

Methodology

The researchers tested two neural TS systems: ACCESS (an aggressive simplifier) and NTS (a conservative simplifier). These systems were applied to two major NLP benchmarks:

  • Relation Extraction (RE): Identifying relationships between entities (e.g., in "Steve Jobs founded Apple," the relation is org:founded_by). The study used the TACRED dataset with LSTM and SpanBERT models.
  • Natural Language Inference (NLI): Determining if a "hypothesis" sentence entails, contradicts, or is neutral toward a "premise." The study used the MNLI dataset with a BERT classifier.

The study compared inference-time simplification (replacing test inputs with simplified versions) against data augmentation (adding simplified versions of training data to the original set). For Relation Extraction, a quality control step ensured that simplified sentences still contained the relevant entities.

Key Results

The experiments revealed that machines and humans "read" differently. Simplification helps humans by reducing cognitive load, but for a machine, simplifying at inference time is often destructive.

The Pitfalls of Inference-Time Simplification

Simplifying text at prediction time caused absolute performance drops of 1.38–2.80% F1 on TACRED and a staggering 11.68–49.55% drop in accuracy on MNLI. This suggests that models rely on specific linguistic features and nuances present in the original text that are stripped away during simplification.

The Benefits of Data Augmentation

In contrast, using TS for data augmentation consistently improved results across the board, regardless of whether the model used an older architecture (LSTM) or a modern transformer (SpanBERT/BERT).

Relation Extraction (TACRED)

On the TACRED dataset, models saw an improvement of 0.7–1.98% F1. As shown in Figure 4, the "Simplified + Original (AD)" configuration consistently outperformed the baseline "Original" training set for both LSTM and SpanBERT architectures.

F1 performance on TACRED for Relation Extraction using various training data augmentations.

Natural Language Inference (MNLI)

On the MNLI dataset, the BERT classifier saw an improvement of 0.50–0.65% in accuracy. Figure 5 illustrates these gains, demonstrating that augmenting the training set with a small percentage of simplified data leads to better generalization on the original test sets.

Accuracy gains on MNLI for Natural Language Inference using data augmentation.

Discussion and Insights

TS augmentation works because it acts as a sophisticated form of semantic paraphrasing. It teaches the model that a core relationship (e.g., "founded by") remains valid whether expressed in a 40-word complex narrative or a 10-word simple statement.

However, information preservation is critical. The authors found that TS is most effective when "critical information"—such as the subject and object in Relation Extraction—is not lost. The benefits were also more pronounced on "harder" tasks like TACRED, which features longer sentences and fewer training examples per class than the massive MNLI dataset.

Limitations and Future Work

A major hurdle remains: Information Loss. Current neural TS models are often "black boxes" that may delete vital information or introduce hallucinations. While simple entity checks worked for Relation Extraction, defining "critical information" for tasks like NLI is more complex. Future research should focus on developing task-specific TS models that preserve the specific features required for downstream NLP tasks and creating more robust automated metrics for semantic identity.

Summary updated

BibTeX
@inproceedings{van2021how,
  author = {Hoang Van and Zheng Tang and Mihai Surdeanu},
  title = {{How May I Help You? Using Neural Text Simplification to Improve Downstream NLP Tasks}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2021},
  doi = {10.18653/v1/2021.findings-emnlp.343},
}
Figure