Neural Versus Non-Neural Text Simplification: A Case Study

Islam Nassar, Michelle Ananda-Rajah, Gholamreza HaffariALTA 2019 · OpenReview

TL;DR

This paper presents a novel rule-based text simplification system that outperforms a state-of-the-art neural system in producing simpler text, particularly for domain-specific tasks like medical reports. The authors demonstrate that a hybrid approach, combining rule-based methods with domain-specific data, can achieve high performance even in low-resource settings.

What Problem Does This Paper Address?

Text simplification aims to make complex text more accessible to a wider audience by reducing linguistic complexity without altering the core meaning. Traditionally, this has been approached using either hand-crafted rules (rule-based methods) or by learning patterns from large parallel corpora (data-driven methods), often framed as a machine translation problem. With the recent success of neural networks in machine translation, researchers are exploring their application to text simplification. However, the authors question whether neural architectures are truly optimal for simplification, given its fundamentally different nature from translation. They aim to compare a robust rule-based system against a leading neural approach, investigating their respective strengths and weaknesses, especially in specialized domains where general-purpose resources are insufficient.

Methods

The paper proposes a modular, non-neural text simplification system with two main components:

  • Lexical Simplification: This module first extracts simpler synonyms for words in the input text using multiple lexical databases (WordNet, Thesaurus, etc.) and a "simplicity criterion" based on corpus frequency. It then uses a language model to select the best synonym substitution that results in the simplest sentence, employing an iterative greedy approach. A 5-gram language model trained on the Simple Wikipedia corpus was found to be most effective.
  • Syntactic Simplification: For this, the authors leverage an existing open-source tool, MUSST (Multilingual Syntactic Simplification Tool). This tool applies a set of general-purpose rules to the dependency parse tree of a sentence to perform operations like splitting clauses and changing passive to active voice.

To adapt the system to domain-specific tasks, particularly medical text, they employ a data-driven approach to enrich the lexical module. This involves training a phrase-based machine translation (PBMT) model on a small parallel corpus of complex-simple medical sentences to extract domain-specific jargon and its simpler equivalents.

Datasets

  • General-purpose Simplification:
    • WikiSmall: A dataset derived from Wikipedia.
  • WikiLarge: A superset of WikiSmall, also from Wikipedia.
  • Both datasets were created by automatically aligning complex and simple sentences from ordinary and simple English Wikipedia articles.
  • Medical Simplification:
    • A newly compiled small parallel corpus of 500 complex-simple medical discharge summary sentences.
  • These sentences were manually simplified by a medical expert to target a Grade 6 reading level (Flesch-Kincaid scale).
  • The corpus was curated to have high lexical diversity to represent the underlying medical reports.

Key Insights and Ideas

  • Rule-based systems can outperform neural systems in simplicity: The paper challenges the assumption that neural networks are universally superior for text simplification, showing that a well-designed rule-based system can produce outputs that are qualitatively and quantitatively simpler.
  • Domain adaptation is crucial for specialized text: General-purpose simplification tools struggle with domain-specific jargon. The proposed hybrid approach, combining rule-based methods with PBMT for jargon extraction, effectively addresses this limitation in low-resource settings.
  • Simplicity vs. Readability Trade-off: The rule-based system prioritizes simplicity, sometimes leading to longer sentences and thus slightly reduced readability (higher FKGL scores) compared to the neural system. This is particularly relevant for medical text, where elaboration is often preferred over deletion.
  • Neural systems may over-rely on deletion: The authors observe that the neural system tends to favor deletion operations, which can sometimes alter the meaning of the original sentence, a critical issue in text simplification.

Key Results

  • General-purpose English: The rule-based system achieved higher SARI scores (a measure of simplification goodness) than the neural system (DRESS) on both WikiLarge and WikiSmall datasets. For WikiLarge, the rule-based system achieved 40.42 SARI compared to DRESS's 37.08.
  • Medical English: The rule-based system demonstrated a significant improvement of 11 points on the SARI simplicity scale and a reduction of 5 grade levels on the FKGL scale compared to the original medical sentences. The DRESS system could not be effectively evaluated on this small medical dataset.
  • Qualitative Analysis: The rule-based system was better at substituting difficult words and splitting complex sentence structures, while the neural system was observed to delete content, sometimes altering meaning.

Limitations and Future Work

  • Readability of Rule-Based System: The rule-based system's tendency to produce longer sentences (due to a lack of deletion operations) leads to slightly lower readability scores (higher FKGL) compared to the neural system. The authors suggest incorporating more deletion rules into the syntactic module to improve readability.
  • Domain-Specific Neural Adaptation: The paper notes that the DRESS system could not be effectively tested on their small medical dataset, highlighting a limitation for neural approaches in low-resource, domain-specific scenarios. Future work could explore more data-efficient neural adaptation techniques.
  • Further Refinement of Hybrid Approach: While successful, the hybrid approach could be further optimized. The authors encourage researchers to address the shortcomings identified in both neural and rule-based systems to advance the field of text simplification.

Summary updated

BibTeX
@inproceedings{nassar2019neural,
  author = {Islam Nassar and Michelle Ananda-Rajah and Gholamreza Haffari},
  title = {{Neural Versus Non-Neural Text Simplification: A Case Study}},
  booktitle = {ALTA},
  year = {2019},
  url = {https://openreview.net/forum?id=Myw3eZn6qEh},
}
Figure