TL;DR
This paper tackles the challenge of neural text simplification for low-resource languages, where large, high-quality datasets are scarce. They propose a novel approach using weakly supervised data augmentation, creating synthetic complex-simple sentence pairs from readily available monolingual data and existing simplification models. This method significantly improves simplification performance on Italian and Spanish compared to baselines, demonstrating the effectiveness of weak supervision in data-scarce scenarios.
What Problem Does This Paper Address?
Neural text simplification (NTS) aims to make text more accessible by reducing its complexity while preserving meaning. While NTS has seen significant advancements, these are largely confined to English due to the availability of large, high-quality datasets like Newsela. For languages with fewer resources, creating such datasets requires extensive manual effort, which is not scalable. Existing rule-based simplification methods, while less data-hungry, lack portability and adaptability. This paper addresses the critical gap of enabling effective NTS for low-resource languages by exploring data augmentation techniques that leverage weak supervision, thereby bypassing the need for massive manually curated datasets.
Methods
The core of this paper lies in its data augmentation strategies for NTS. They build upon the standard attentional encoder-decoder model, extended with a pointer-generator network, which allows for both copying words and generating new ones. The novel contributions are in how they augment the training data:
- Oversampling: A straightforward approach where existing gold-standard training pairs are simply repeated multiple times to increase dataset size.
- Simple-to-Simple Synthetic Pairs: This involves extracting simple sentences from large monolingual corpora using readability heuristics. These extracted simple sentences are then duplicated to create synthetic pairs. The intuition is to enrich the training data with more examples of simple language, potentially improving word embeddings and biasing the decoder towards simpler outputs.
- Simple-to-Complex Synthetic Pairs: This is the most innovative technique. They train a "complexifier" system (using a standard MT approach) on existing gold data. This system is then used to artificially complicate the simple sentences extracted in the previous step. The resulting pairs are then inverted to create synthetic complex-to-simple training data. While the "complex" side might be of lower quality, the authors hypothesize that the human-generated "simple" side is preserved, which is crucial for training the simplification model. This approach is inspired by similar techniques in machine translation and keyword-to-question generation.
- Pre-trained Embeddings: They also investigate the impact of using pre-trained word embeddings from large monolingual corpora, hypothesizing they might be more beneficial in low-resource settings.
Datasets
The experiments are conducted on Italian and Spanish, chosen for their limited availability of high-quality simplification data.
- Italian Gold Standard: A combination of three datasets: SIMPITIKI (manually curated), Brunato et al. (manually curated), and a subset of PaCCSS-it (automatically extracted from the web, filtered for quality). This resulted in approximately 32,210 complex-to-simple pairs.
- Italian Simple Monolingual Data: Extracted from a large Italian monolingual corpus (including OpenSubtitles, Paisà corpus, Wikipedia, and legal texts), keeping the 500,000 most readable sentences based on readability metrics.
- Spanish Gold Standard: The Spanish Newsela corpus, containing 1,221 documents, aligned into complex-simple pairs using the CATS-Align tool, resulting in 55,890 pairs after filtering.
- Spanish Simple Monolingual Data: Extracted from a corpus of simple Spanish texts for language learners, ranking sentences by the Flesch-Szigriszt readability score, yielding 484,325 simple sentences.
- Pre-trained Embeddings: Word embeddings were trained on large monolingual corpora for both languages. For Spanish, this included the Spanish Billion Word Corpus augmented with their extracted simple Spanish texts.
Key Insights and Ideas
- Weak Supervision is Powerful: The most significant finding is that weakly supervised data augmentation, particularly the simple-to-complex synthetic pairs, can dramatically improve NTS performance in low-resource settings, even when the synthetic "complex" sentences are imperfect.
- Simple-to-Simple Pairs Can Be Harmful: Counter-intuitively, the simple-to-simple synthetic pairs sometimes degraded performance, especially for Spanish. The authors suggest this might push the model towards simply copying the source sentence rather than genuinely simplifying it, leading to instability.
- Pre-trained Embeddings Offer a Boost: Using pre-trained word embeddings consistently improved performance, providing valuable linguistic information that might not be captured by small, task-specific datasets.
- Language Differences Matter: The paper notes that Italian models generally performed better than Spanish ones, potentially due to differences in the quality and nature of the gold standard data and the methods used for extracting simple monolingual sentences.
Key Results
The paper evaluates performance using the SARI (Systeme d'Annotation et de Révision Interactive) score.
- Simple-to-Complex Pairs Lead: The simple-to-complex synthetic pairs, both alone and combined with pre-trained embeddings, yielded the largest performance gains, outperforming the baseline by up to +3.4 SARI points.
- Best Configuration: The best results were achieved by multiplying the training data by 10 and adding the simple-to-complex synthetic data. This configuration outperformed the baseline by +5.4 SARI points for Italian and +2.4 for Spanish.
- Comparison to Rule-Based: The proposed neural models, even in low-resource conditions, significantly outperformed a rule-based system (MUSST) on the same test set, demonstrating the continued promise of neural approaches.
Limitations and Future Work
- Quality of Synthetic Data: The authors acknowledge that the simple-to-complex synthetic pairs are of lower quality, which is a trade-off for data availability.
- Simple-to-Simple Pair Issues: The negative impact of simple-to-simple pairs on performance highlights a need for more nuanced data augmentation strategies.
- Language-Specific Nuances: The observed performance differences between Italian and Spanish suggest that further investigation into language-specific data characteristics and extraction methods is warranted.
Future work includes extending experiments to more languages, exploring additional data augmentation techniques (e.g., using machine translation for cross-lingual gold data generation, bootstrapping), and refining the simple-to-simple pair creation process.
Summary updated