Sentence Simplification Capabilities of Transfer-Based Models

Sanja Stajner, Kim Cheng Sheang, Horacio SaggionAAAI 2022 · DOI

Abstract

This paper presents new state-of-the-art sentence simplification (SS) systems for English and Spanish by fine-tuning Transformer models, such as T5 and mBART, with specific control tokens. Moving beyond standard automated metrics, the authors propose a rigorous expert evaluation framework based on established "easy-to-read" guidelines. The results indicate that these models not only reduce sentence length but significantly improve accessibility while maintaining high levels of meaning preservation compared to previous neural approaches.

Introduction

Sentence Simplification is a vital accessibility tool for the significant portion of the population—estimated at 50% in some highly developed countries—who struggle with complex syntax and advanced vocabulary. While modern neural systems treat simplification as a translation task (Complex $\rightarrow$ Simple), they often function as "black boxes." These models may produce shorter sentences that lose essential meaning or fail to adhere to actual accessibility standards, such as avoiding passive voice or hidden verbs. Furthermore, standard metrics like SARI or BLEU often fail to capture whether a system truly assists readers with low literacy. This research aims to bridge the gap between powerful AI models and the practical needs of human readers.

Methodology: Controllable Simplification and Expert Evaluation

The study employs a transfer-based approach, fine-tuning large pre-trained language models: T5 for English, and mBART and mT5 for Spanish. To guide the simplification process, the researchers introduced four control tokens during training:

  • Compression (C): Specifies the desired output length.
  • Levenshtein similarity (L): Controls the degree of structural change.
  • Word Rank (WR): Targets the desired level of vocabulary (lexical) complexity.
  • Dependency Tree Depth (DTD): Targets the desired level of grammatical (syntactic) complexity.

A primary contribution of this work is the shift toward expert human evaluation. Rather than relying solely on crowdsourced opinions, linguistic experts assessed model outputs against 16 specific easy-to-read rules.

Table 2: 16 rules for simpler forms used in expert evaluation.

Results and Analysis

The proposed T5-base model set a new state-of-the-art for English, achieving a SARI score of 44.9 on the ASSET dataset, surpassing previous benchmarks like MUSS (43.6). For Spanish, the mBART model reached a SARI of 37.1, establishing a new performance standard for the language.

Expert trials highlighted a critical "meaning vs. simplicity" trade-off. While previous models often achieved simplicity by deleting essential information, the T5-base model was significantly more effective at keeping the core meaning intact. It achieved a meaning preservation score of 4.2/5 and a simplicity score of 4.3/5 in head-to-head trials.

Table 5: Examples of automatic simplification transformations by T5-base and MUSS-sup.

The models successfully performed complex operations, such as converting passive voice to active and splitting long, "run-on" sentences into clearer, individual units. However, the Spanish models were notably more conservative and less diverse in their transformations, likely due to the smaller training set (7.4k pairs compared to 296k for English).

Limitations and Future Work

Despite the improvements, the models occasionally "hallucinate" information, such as incorrectly expanding proper nouns. Additionally, the "Spanish gap" remains a challenge; the smaller training datasets and the use of multilingual models may not yet capture the full nuances of the language as effectively as monolingual English models. Future research directions include moving toward "document-level" simplification to ensure that simplifying individual sentences does not disrupt the overall flow or context of an article.

Summary updated

BibTeX
@inproceedings{stajner2022sentence,
  author = {Sanja Stajner and Kim Cheng Sheang and Horacio Saggion},
  title = {{Sentence Simplification Capabilities of Transfer-Based Models}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2022},
  doi = {10.1609/aaai.v36i11.21477},
}
Figure