Abstract: While paraphrasing is critical both for interpretation and generation of natural language, current systems use manual or semi-automatic methods to collect paraphrases. We present an unsupervised learning algorithm for identification of paraphrases from a corpus of multiple English translations of the same source text. Our approach yields phrasal and single word lexical paraphrases as well as syntactic paraphrases.
Introduction
In Natural Language Processing (NLP), the ability to identify and generate paraphrases is essential for tasks such as text summarization and fluent natural language generation. Traditional approaches to building paraphrase lexicons have largely relied on manual curation or lexical databases like WordNet. However, these methods suffer from limited scope—often restricted to single-word synonyms—and poor scalability across different domains.
To address these limitations, researchers at Columbia University developed an unsupervised learning algorithm that automatically extracts paraphrases by analyzing multiple English translations of the same source text. By leveraging the natural variation in how different translators render the same meaning, the system identifies equivalent expressions ranging from single words to complex syntactic patterns.
Methodology: Unsupervised Co-Training
The core of the approach is an unsupervised co-training algorithm based on the DLCoTrain model. The process follows a bootstrapping cycle:
- Sentence Alignment: The system aligns sentences from different translations of the same literary work (e.g., multiple versions of Madame Bovary).
- Seeding and Context Extraction: It begins with a "seed" of identical words found in aligned sentences and analyzes the surrounding contexts (left and right word sequences or Part-of-Speech tags).
- Pattern Expansion: If a specific context consistently surrounds identical words, it is marked as a "good predictor." The algorithm then identifies different phrases appearing in these predictive environments as potential paraphrases.
- Syntactic Rule Induction: Beyond simple word swaps, the system identifies morpho-syntactic rules. For example, it can learn that the possessive construction
[Noun1] 's [Noun2]is equivalent to the prepositional phrase[Noun2] of the [Noun1].
Experimental Setup and Results
The researchers utilized a "monolingual parallel corpus" consisting of 11 English translations of 5 classic foreign novels, resulting in 44,562 aligned sentence pairs. This literary corpus presents a significant challenge due to the creative liberties taken by translators, making it "noisier" than standard bilingual corpora.
The algorithm successfully produced 9,483 pairs of lexical paraphrases and 25 morpho-syntactic rules. Human evaluation demonstrated a high level of accuracy, with a precision of 91.8% when paraphrases were judged within their original sentence contexts. Furthermore, the system significantly outperformed standard Machine Translation (MT) baselines, achieving 71.6% precision compared to the MT baseline's 52.7% on a specific test set. Notably, nearly 30% of the identified paraphrases were multi-word phrases.
Insights into Semantic Variation
A key finding of the study is that paraphrasing extends far beyond simple synonymy. By comparing the extracted pairs to WordNet, the authors found that only 35% were direct synonyms. The remaining pairs included hyperonyms, siblings in a hierarchy, or context-specific equivalents.
This suggests that the algorithm captures how humans naturally vary language to convey the same underlying meaning, even when the words themselves are not strictly synonymous in a general dictionary sense.
Conclusion and Future Work
While the system demonstrates high precision, measuring "recall" remains a challenge due to the lack of a comprehensive gold-standard list. Additionally, the use of 19th-century literature results in a somewhat dated vocabulary. Future research aims to apply this methodology to contemporary "comparable" corpora, such as news reports from different agencies on the same events, and to incorporate more sophisticated syntactic parsers to further refine context representation.
Summary updated