TL;DR
This paper introduces a probabilistic model to assess how well automatically generated paraphrases of predicate phrases are semantically equivalent, substitutable, and grammatical. By combining a structured N-gram language model for grammaticality with a distributional similarity measure for semantic equivalence, the model aims to provide a more comprehensive evaluation than existing methods.
What Problem Does This Paper Address?
Generating and recognizing paraphrases is crucial for many Natural Language Processing (NLP) tasks, such as aggregating opinions from web data or suggesting alternative phrasing for document writing. A key challenge is acquiring comprehensive paraphrase knowledge. Previous work often focused on static, lexicalized pairs or abstract patterns, leading to limited coverage. This paper specifically tackles the problem of evaluating automatically generated paraphrases, particularly for predicate phrases in Japanese, by proposing a model that considers three essential conditions: semantic equivalence, contextual substitutability, and grammaticality.
Methods
The core of this paper is a probabilistic model designed to compute the likelihood that a given pair of phrases (s, t) are paraphrases. This model is formulated as a conditional probability $P(t|s)$, which is decomposed into two factors:
- Grammaticality Factor ($P(t)$): This component uses a structured N-gram language model to assess how grammatical the target phrase $t$ is. For Japanese, where word order can be flexible, they employ dependency structures (either morpheme-based or content-function-based) to build the language model.
- Similarity Factor ($Sim(s, t)$): This factor estimates semantic equivalence and substitutability by comparing contextual features of the two phrases. Two feature sets are explored:
- BOW (Bag-of-Words): Assesses similarity based on co-occurring words in the same sentence.
- MOD (Modifier/Modifiee): Assesses substitutability by looking at shared modifiers and modifiees.
The model is asymmetric, meaning $P(t|s)$ is not necessarily equal to $P(s|t)$, reflecting that one phrase might be a better paraphrase of the other.
Datasets
The experiments were conducted using a test suite developed in a previous work (Fujita and Sato, 2008), consisting of 176,541 paraphrase candidates generated for 4,002 high-frequency phrases from a newspaper corpus. For training the language models and estimating feature probabilities, they utilized:
- Mainichi Newspaper Corpus: A 15-year corpus (21 million sentences) used for building structured N-gram language models.
- Web Corpus (WebCP): A large corpus of 470 million sentences collected from the Web, used for estimating feature probabilities.
Key Insights and Ideas
- The Importance of Grammaticality: The paper strongly emphasizes that simply having semantically similar phrases isn't enough; they must also be grammatically sound. This is a critical aspect often overlooked in purely similarity-based approaches.
- Structured Language Models for Grammaticality: Using dependency structures (like CFDS) for language modeling proves more effective than simpler morpheme-based structures, especially for languages like Japanese with flexible word order.
- Feature Sets Matter: While BOW is a basic approach, the MOD feature set (focusing on modifiers and modifiees) shows promise for capturing substitutability, and combining features (HAR) can sometimes yield better results.
- Web Data Nuances: Surprisingly, the massive Web corpus didn't consistently outperform the more curated newspaper corpus for feature estimation, suggesting that the quality and noise in web data can be a significant factor.
- Limitations of Existing Baselines: The paper highlights that even established measures like ParLin and Parskew, which focus solely on similarity, are outperformed by their proposed model when grammaticality is considered.
Key Results
- Overall Performance: The proposed probabilistic model, when considering both grammaticality and similarity, generally outperformed existing similarity-only measures (ParLin, Parskew) in terms of precision for judging paraphrase quality.
- Grammaticality Model: The CFDS (Content-Function-based Dependency Structure) language model consistently outperformed the MDS (Morpheme-based Dependency Structure) model for assessing grammaticality.
- Feature Sets: For similarity, MOD and HAR (harmonic mean of BOW and MOD) generally performed better than BOW alone.
- Corpus Impact: The Mainichi corpus often yielded better results than the larger WebCP for feature estimation, possibly due to parsing errors on web data.
- Snippet Count: Increasing the number of web snippets used for feature estimation (from 500 to 1,000) did not lead to significant performance improvements, likely due to the limited number of expressions for which enough snippets could be retrieved.
Limitations and Future Work
- Model Performance: The paper acknowledges that while their model shows promise, it did not always outperform existing models, particularly when considering the strict precision metric. The authors suggest that a more sophisticated language model is needed.
- Feature Extraction Noise: The performance of feature extraction from web data can be affected by noise and errors from morphological and dependency parsers, especially when these parsers are tuned for different domains (e.g., newspaper vs. general web).
- Future Directions:
- Develop more sophisticated structured language models (e.g., using latent classes or incorporating external resources like Japanese functional expression dictionaries) to improve grammaticality assessment.
- Conduct experiments on template-like paraphrase knowledge beyond just productive paraphrases.
- Investigate the trade-off between the number of web snippets and performance, and the impact of snippet quality.
Summary updated