Unsupervised Text Simplification Without Parallel Data

28 concepts18 papers1 proposal

Automatic text simplification aims to make text easier to read while preserving its original meaning. Traditionally, this task has been treated similarly to machine translation: models are trained on large, human-annotated parallel corpora containing aligned "complex-to-simple" sentence pairs (such as alignments between standard English Wikipedia and Simple English Wikipedia).

However, human-annotated parallel data is rare, expensive to produce, and virtually non-existent for low-resource languages or specialized domains like medicine. To bypass this bottleneck, a powerful research direction has emerged: unsupervised text simplification. By framing simplification as an optimization problem balancing fluency, simplicity, and meaning preservation, these systems achieve human-like simplifications using only unaligned text or pre-trained language models.


The Three Core Forces of Simplification

To understand why unsupervised text simplification works, we must look at it as a balancing act among three competing constraints:

  1. Fluency (Grammaticality): Is the output natural language?
  2. Meaning Preservation (Retention): Does the simplified text convey the same core information?
  3. Simplicity (Readability): Is the vocabulary and sentence structure easier to comprehend?

Supervised models learn to balance these forces implicitly from human examples. Unsupervised models, by contrast, must decouple these forces and optimize them explicitly. They achieve this through three main paradigms: harvesting pseudo-parallel data, context-aware lexical substitution, and iterative search-based editing.


Harvesting Pseudo-Parallel Corpora

One way to eliminate the need for manual annotation is to build parallel datasets automatically from raw text. Instead of relying on humans to write simple versions of complex documents, algorithms can search massive collections of unaligned text to discover naturally occurring pairs of varying complexity.

[Raw Web Crawl / Unaligned Corpora]
               │
               ▼
   [Sentence Similarity Scoring]  ◄─── (Word Embeddings / Contextual Vectors)
               │
               ▼
 [Automatically Aligned Pairs]   ───► [Train Readability Estimators & Generators]

Tomoyuki Kajiwara et al. in Building a Monolingual Parallel Corpus for Text Simplification Using Sentence Similarity Based on Alignment between Word Embeddings demonstrated that word embeddings can be used to compute semantic similarity between sentences in unaligned corpora. By automatically pairing complex sentences with simpler sentences that share an informational core, they constructed high-quality parallel corpora without human intervention. Similarly, Dominique Brunato et al. (PaCCSS-IT: A Parallel Corpus of Complex-Simple Sentences for Automatic Text Simplification) showed that these complex-simple pairs can be successfully harvested from general-purpose web crawls, bypassing specialized platforms like Wikipedia entirely.

Once these pairs are harvested, they can be used to train other components of a simplification pipeline. For instance, Rina Miyata et al. (Unsupervised Sentence Readability Estimation Based on Parallel Corpora for Text Simplification) utilized the inherent complexity differences in automatically harvested sentence pairs to train readability estimators, learning relative difficulty without requiring absolute, expert-annotated readability scores.


Context-Aware Lexical Substitution

Lexical simplification focuses on replacing difficult words with easier synonyms. Early unsupervised approaches, such as the one proposed by Or Biran et al. (Putting it Simply: a Context-Aware Approach to Lexical Simplification), relied on the statistical differences between unaligned corpora to identify when to swap a word. Later, Goran Glavaš et al. (Simplifying Lexical Simplification: Do We Need Simplified Corpora?) and Gustavo H. Paetzold et al. (Unsupervised Lexical Simplification for Non-Native Speakers) introduced word embeddings and domain-specific frequency data to find substitutes, drastically increasing text coverage without relying on hand-crafted lexicons like WordNet.

The intuition behind modern lexical substitution reaches its peak with pre-trained language models. In Lexical Simplification with Pretrained Encoders, Jipeng Qiang et al. introduced BERT-LS, a method that frames lexical simplification as a masked language modeling task.

Original:  The cat slept on the [gargantuan] sofa.
Masked:    The cat slept on the [MASK] sofa.
BERT Proposes:  [large (prob: 0.45), huge (prob: 0.30), big (prob: 0.15)]

By masking the complex word, the pre-trained model naturally proposes substitutes that are grammatically fluent and semantically appropriate for the surrounding context. Because language models are exposed to massive amounts of raw text, they inherently assign higher probabilities to more common, simpler words. This allows the system to simultaneously optimize for context preservation and simplicity without ever seeing a parallel corpus.


Simplification as Iterative Search and Editing

When moving from single words to full sentences, simplification requires structural changes like phrase deletion and syntactic restructuring. Unsupervised systems handle this by framing sentence simplification as a discrete optimization or guided search problem.

Instead of generating a sentence from scratch, these models start with the original complex sentence and apply a series of edit operations (such as word substitution or phrase deletion). Sigrid Klerke et al. (Simple, readable sub-sentences) pioneered this by randomly sampling sub-sentences via deletion and ranking them using a readability-informed loss function.

This concept was expanded into iterative generation and revision frameworks. Dhruv Kumar et al. (Iterative Edit-Based Unsupervised Sentence Simplification) and Mohammad Dehghan et al. (GRS: Combining Generation and Revision in Unsupervised Sentence Simplification) developed systems that use a "complex component detector" to guide discrete edit operations. The model proposes edits, and a multi-factor scoring function evaluates whether the edit improved simplicity while maintaining fluency and meaning.

To make these edits highly customizable, researchers have integrated reinforcement learning and controllable generation. Guanlin Li et al. (Aligning Sentence Simplification with ESL Learner's Proficiency for Language Acquisition) combined large language models with structured reward functions and lookahead searches to tailor text to specific language proficiency levels. Similarly, Louis Martin et al. (MUSS: Multilingual Unsupervised Sentence Simplification by Mining Paraphrases) and Julius Monsen et al. (Controllable Sentence Simplification in Swedish Using Control Prefixes and Mined Paraphrases) utilized web-mined paraphrases alongside explicit control prefixes (e.g., length and complexity tokens) to guide the generation process.


Why the Unsupervised Approach Succeeds

Unsupervised text simplification works because it leverages the massive, implicit linguistic knowledge already present in unannotated text and pre-trained models.

  • Robust Domain Adaptation: Supervised models are often bottlenecked by the narrow domain of their training data (e.g., Wikipedia). Unsupervised frameworks can easily adapt to niche domains. For example, Tarek Sakakini et al. (Context-Aware Automatic Text Simplification of Health Materials in Low-Resource Domains) successfully simplified complex medical jargon by anchoring an unsupervised system to specialized medical ontologies and patient-centric language models.
  • Style Disentanglement: Rather than memorizing specific human rewrites, unsupervised neural architectures can explicitly separate style from content. Sai Surya et al. (Unsupervised Neural Text Simplification) demonstrated that using a shared encoder with dual decoders and adversarial training on attention mechanisms allows a model to capture the stylistic patterns of simple language from non-aligned corpora, applying those patterns to complex text while keeping the underlying meaning intact.

By treating simplicity as a measurable direction in a semantic space rather than a translation task, unsupervised text simplification proves that high-quality, fluent, and accessible text transformation does not require human-aligned training pairs.

Key Foundational Work

Simple, readable sub-sentences (2013) — contributed 2 concept elements to this direction.
Putting it Simply: a Context-Aware Approach to Lexical Simplification (2011) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_unsupervised_lexical_simplification_637e81fc,
  title = {unsupervised lexical simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-lexical-simplification/rdr_637e81fcfd424c958bf2cf2b33680fba},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_unsupervised_lexical_simplification_637e81fc_v1,
  title = {unsupervised lexical simplification},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/unsupervised-lexical-simplification/rdr_637e81fcfd424c958bf2cf2b33680fba/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake