TL;DR
This paper introduces an unsupervised method to automatically build monolingual parallel corpora for text simplification by leveraging word embeddings and sentence similarity. This approach significantly reduces the human annotation cost typically required for such datasets, enabling more robust text simplification models.
What Problem Does This Paper Address?
Text simplification aims to rewrite complex text into a simpler form while preserving meaning, benefiting language learners and children. A key component for training statistical machine translation (SMT) models for text simplification is a monolingual parallel corpus – pairs of complex and simple sentences. However, creating these corpora is labor-intensive and costly due to the need for human annotation. While parallel corpora exist for a limited number of languages, they are scarce and often not publicly available, hindering research progress. This paper addresses the challenge of efficiently and automatically constructing these valuable datasets.
Methods
The core of this work is an unsupervised method for building monolingual parallel corpora using sentence similarity based on word embeddings. The process involves:
- Comparable Corpus as Input: The method starts with a comparable corpus, which contains texts on similar topics but not necessarily direct sentence-to-sentence translations or simplifications.
- Sentence Similarity Calculation: The authors propose four types of sentence similarity measures, all based on aligning word embeddings between sentences. These include:
- Average Alignment: Averages the similarity of all possible word pairs between two sentences.
- Maximum Alignment: Focuses on the most similar word in the target sentence for each word in the source sentence, then averages these maximum similarities. This is a many-to-one alignment.
- Hungarian Alignment: Uses the Hungarian algorithm to find a one-to-one word alignment that maximizes the sum of word similarities.
- Word Mover's Distance (WMD): Treats sentences as distributions of words and calculates the minimum "distance" to transform one distribution into another.
- Pair Extraction: Sentences are paired if their computed similarity exceeds a predefined threshold.
The authors found that Maximum Alignment performed best for their task, as it effectively handles many-to-one relationships, which are common in text simplification where a single word or phrase in a simple sentence might correspond to multiple words or a complex phrase in the original.
Datasets
- Benchmark Dataset: The authors used a benchmark dataset created by Hwang et al. (2015) for evaluating sentence similarity measures. This dataset consists of sentence pairs extracted from English Wikipedia and Simple English Wikipedia, labeled with four categories: Good (G), Good Partial (GP), Partial, and Bad. For their classification task, they defined "parallel" sentences as either G (synonymous) or G+GP (synonymous or entailing).
- English Wikipedia and Simple English Wikipedia: These were used as the source for constructing the monolingual parallel corpus for text simplification. The method paired articles by title and then extracted sentences.
Key Insights and Ideas
- Many-to-One Alignment is Crucial: The success of "Maximum Alignment" highlights that text simplification often involves mapping multiple words or complex phrases in a source sentence to a single, simpler word or phrase in the target. Simple one-to-one alignment methods (like Hungarian Alignment) can be too restrictive.
- Word Embeddings Bridge Semantic Gaps: Using word embeddings allows the system to capture semantic similarity between words that are not lexically identical, which is essential for identifying paraphrases and simplifications.
- Unsupervised Approach is Viable: The paper demonstrates that it's possible to build high-quality parallel corpora without expensive human annotation, significantly lowering the barrier to entry for text simplification research.
- Simple English Wikipedia is a Good Source: The readability score distribution analysis (Figure 2) supports the intuition that Simple English Wikipedia contains genuinely simpler language, making it a valuable resource for contrastive analysis.
Key Results
- Improved Binary Classification: The proposed Maximum Alignment method achieved a higher F1 score (0.638) in classifying parallel vs. nonparallel sentences compared to previous methods (e.g., Zhu et al. at 0.550, Coster and Kauchak at 0.564, Hwang et al. at 0.712, but the paper's method achieved 0.717 for G vs. O and 0.638 for G+GP vs. O).
- State-of-the-Art Text Simplification: An SMT model trained on the automatically constructed corpus achieved a BLEU score of 47.5, which was 3.2 points higher than a model trained on a state-of-the-art existing corpus (44.3).
- High-Quality Corpus Construction: The method successfully built a large monolingual parallel corpus (492,993 sentence pairs) from English Wikipedia and Simple English Wikipedia. Examples show it captures synonymous expressions and sentence deletions effectively.
Limitations and Future Work
- Language Dependency: While the method is unsupervised, the quality of word embeddings can be language-dependent. The paper focuses on English.
- Comparable Corpus Requirement: The method relies on having a comparable corpus available. Building such a corpus for many languages might still be a challenge.
- Future Work: The authors plan to extend this approach to build monolingual parallel corpora from raw text by combining their sentence similarity measure with readability assessments. They also aim to apply this to more languages and explore new opportunities in text simplification.
Summary updated