TL;DR
This paper introduces a minimally supervised method for extracting paraphrases across multiple languages from web-based definition sentences. By leveraging Wikipedia definitions and a novel similarity measure, the approach achieves competitive results with state-of-the-art supervised methods while requiring significantly less manual annotation.
What Problem Does This Paper Address?
Automatic paraphrase extraction is crucial for various Natural Language Processing (NLP) tasks, such as question answering, information retrieval, and text summarization. Existing methods often rely on large amounts of manually annotated data or parallel corpora, which are expensive and time-consuming to create, and can be language-dependent. The authors aim to bridge this gap by developing a method that can extract paraphrases with high precision and recall across different languages using minimal supervision, specifically by exploiting the inherent paraphrasing found in definition sentences on the web.
Methods
The proposed method consists of two main stages: definition extraction and paraphrase extraction.
Definition Extraction:
* Minimally Supervised Approach: Instead of relying on manually labeled definition sentences, the authors construct a training dataset (TrDat) automatically. Positive examples (Pos) are the first sentences of Wikipedia articles, assumed to be definitions. Negative examples (Neg) are randomly sampled sentences from the web.
* Pattern Mining: The system identifies characteristic patterns (N-grams, subsequences, dependency subtrees) that appear frequently in definition sentences but rarely in non-definition sentences. These patterns are used to train a classifier.
* SVM Classifier: A Support Vector Machine (SVM) classifier is trained using these patterns to distinguish definition sentences from non-definition sentences.
* Term Identification: The method also identifies the "defined term" within a sentence, which is crucial for pairing definition sentences that define the same concept.
Paraphrase Extraction:
* Coupling Definition Sentences: Definition sentences with the same defined term are paired.
* Phrase Extraction: Predicate phrases are extracted from these coupled definition sentences.
* Similarity Measures: Two novel similarity measures are introduced to score candidate phrase pairs:
* Local Similarity (localSim): A weighted sum of 37 subfunctions that capture similarities between the phrases themselves and their immediate contexts. This is inspired by previous work but extended.
* Global Similarity (globalSim): A novel measure that considers the semantic similarity of the differences between two phrases across all extracted definition pairs. The intuition is that if the differing parts of two phrases have the same meaning, they are more likely to be paraphrases.
* Scoring and Ranking: A final score is computed by combining localSim and globalSim, which is then used to rank paraphrase candidates.
Datasets
- Training Data (
TrDat): Constructed automatically using: - Positive Examples (
Pos): First sentences of Wikipedia articles (English, Japanese, Chinese). - Negative Examples (
Neg): Randomly sampled sentences from large web corpora (Japanese Web pages, ClueWeb09 for English and Chinese). - Evaluation Data:
- Definition Extraction: 10% of the web corpus for each language. For comparison, the WCL datasets (Navigli and Velardi, 2010) were used for English.
- Paraphrase Extraction: Definition pairs extracted from the 10% web corpus. For comparison, the supervised method by Hashimoto et al. (2011) was used for Japanese.
Key Insights and Ideas
- Wikipedia as a Rich Source: The first sentence of a Wikipedia article is a surprisingly reliable indicator of a definition, providing a large, readily available source of positive training data for definition extraction.
- Leveraging Definition Structure: The core idea that definition sentences for the same concept often contain paraphrases is powerful and can be exploited across languages.
- Minimally Supervised is Viable: The success of the method demonstrates that high-quality paraphrase extraction can be achieved with significantly less manual annotation than previously thought, making it more scalable.
- Global Context Matters for Paraphrases: The novel
globalSimmeasure highlights that the broader context in which phrase differences appear is a strong indicator of their semantic equivalence, going beyond local phrase similarity. - Language Independence: The method's design, particularly the pattern mining and similarity measures, shows a high degree of language independence, working effectively for English, Japanese, and Chinese.
Key Results
- Definition Extraction: The proposed method achieved competitive results with state-of-the-art supervised methods for English definition extraction. For multilingual definition extraction, it extracted millions of definitions with reasonable precision (e.g., 70% for English, 62.5% for Japanese, 67% for Chinese).
- Paraphrase Extraction:
- Outperformed unsupervised methods (Paşca and Dienes, 2005; Koehn et al., 2007) for English, Japanese, and Chinese.
- Was comparable to the state-of-the-art supervised method for Japanese (Hashimoto et al., 2011).
- Achieved high precision (e.g., 92% for English, 82.5% for Japanese, 82% for Chinese) for extracted paraphrases.
- Demonstrated the effectiveness of the
globalSimmeasure, as removing it (Proposedlocal) led to a performance drop.
Limitations and Future Work
- Precision vs. Recall Trade-off: While achieving high precision, the method might extract fewer "content word variation" paraphrases compared to some other methods, potentially impacting recall for certain types of paraphrases.
- Filtering Heuristics: The filtering of defined terms appearing more than 1,000 times is a heuristic.
- Future Work: The authors mention investigating a more mathematically well-motivated model for
globalSimand exploring other ways to combine similarity functions. The language independence suggests potential for application to many more languages.
Summary updated