Divide-and-Conquer Text Simplification by Scalable Data Enhancement

Sanqiang Zhao, Rui Meng, Hui Su, Daqing He2022 · DOI

Abstract

Text simplification is a task to reduce the complexity of a text while retain its original meaning. It can facilitate people with low-literacy skills or language impairments, such as children and individuals with dyslexia and aphasia, to read and understand complicated materials. Normally, substitution, deletion, reordering, and splitting are considered as four core operations for performing text simplification. Thus an ideal model should be capable of executing these operations appropriately to simplify a text. However, by examining the degree that each operation is exerted in different datasets, we observe that there is a salient discrepancy between the human annotation and existing training data that is widely used for training simplification models. To alleviate this discrepancy, we propose an unsupervised data construction method that distills each simplifying operation into data via different automatic data enhancement measures. The empirical results demonstrate that the resulting dataset SimSim can support models to achieve better performance by performing all operations properly.

The Distribution Discrepancy Problem

Effective text simplification relies on four distinct linguistic operations: substitution (swapping complex words for simpler ones), deletion (removing non-essential information), reordering (simplifying syntax), and splitting (breaking long sentences into multiple shorter ones).

A major hurdle in training simplification models is the "distribution discrepancy." Most existing models rely on large-scale datasets like WikiLarge or Newsela, which are generated through automatic alignment of complex and simple articles. However, these datasets often exhibit a heavy bias toward deletion while neglecting other critical operations like splitting and substitution. When models trained on these biased datasets are evaluated against high-quality, human-annotated benchmarks like Turk or ASSET, they perform poorly because they have not learned the full range of strategies humans employ.

Distribution of simplification operations across various datasets, highlighting the discrepancy between standard training sets and human-annotated benchmarks.

The SimSim Pipeline: A Divide-and-Conquer Approach

To address this discrepancy, the authors introduce SimSim, an unsupervised data enhancement pipeline designed to explicitly simulate core simplification operations. The construction process involves several stages:

  1. Seed Selection and Back-Translation: High-quality sentence pairs are filtered using BERTScore and then expanded through back-translation into 103 pivot languages to increase linguistic diversity.
  2. Quality Control: To mitigate "translationese" or hallucinations common in synthetic data, GPT-2 is used to rank and filter candidates based on naturalness.
  3. Operation Simulation: Specific operations are injected into the data:
  4. Substitution: Applying rules from the Paraphrase Database (PPDB).
  5. Dropping: Randomly removing prepositional, adjective, or adverb phrases to simulate concise editing.
  6. Splitting: Incorporating examples from the WikiSplit dataset to teach the model how to break down complex syntax.

Key Results and Insights

Models trained on the full SimSim pipeline (SimSim-S3) significantly outperformed baselines on the ASSET benchmark, achieving a SARI score of 52.37 compared to 50.79 for Wiki-Auto.

One of the most notable successes was in sentence splitting. Traditional datasets like Newsela almost entirely lack splitting operations, whereas SimSim-S3 successfully matches the distribution found in human-annotated data. As shown in the table below, SimSim achieves a splitting proportion of 0.399, closely aligning with the 0.310 proportion found in the gold-standard ASSET dataset.

Comparison of sentence splitting proportions across different corpora.

Conclusion

The SimSim framework demonstrates that synthetic data, when carefully constructed to simulate specific linguistic transformations, can be more effective than noisy real-world data. By balancing the four core simplification operations, the model moves beyond simple summarization toward true text simplification. Future work may expand this "divide-and-conquer" approach to document-level simplification, where maintaining context across multiple sentences remains a challenge.

Summary updated

Abstract

Text simplification is a task to reduce the complexity of a text while retain its original meaning. It can facilitate people with low-literacy skills or language impairments, such as children and individuals with dyslexia and aphasia, to read and understand complicated materials. Normally, substitution, deletion, reordering, and splitting are considered as four core operations for performing text simplification. Thus an ideal model should be capable of executing these operations appropriately to simplify a text. However, by examining the degree that each operation is exerted in different datasets, we observe that there is a salient discrepancy between the human annotation and existing training data that is widely used for training simplification models. To alleviate this discrepancy, we propose an unsupervised data construction method that distills each simplifying operation into data via different automatic data enhancement measures. The empirical results demonstrate that the resulting dataset SimSim can support models to achieve better performance by performing all operations properly.
BibTeX
@misc{zhao2022divide,
  author = {Sanqiang Zhao and Rui Meng and Hui Su and Daqing He},
  title = {{Divide-and-Conquer Text Simplification by Scalable Data Enhancement}},
  year = {2022},
  doi = {10.18653/v1/2022.tsar-1.15},
}
Figure