'John ate 5 apples' != 'John ate some apples': Self-Supervised Paraphrase Quality Detection for Algebraic Word Problems

Rishabh Gupta, Venktesh V, Mukesh K. Mohania, Vikram Goyal

TL;DR

This paper introduces ParaQD, a novel self-supervised method for detecting the quality of paraphrased algebraic word problems. It addresses the critical issue of paraphrasers losing essential information, making problems unsolvable, and achieves state-of-the-art performance by outperforming existing methods by up to 32%.

What Problem Does This Paper Address?

In educational settings, paraphrasing algebraic word problems (AWPs) is crucial for generating diverse assessment questions and ensuring students understand the underlying concepts rather than memorizing solutions. However, current paraphrase generation models often fail to preserve critical information like numbers and units, rendering the paraphrased problems unsolvable or semantically incorrect. This necessitates an automated way to evaluate the quality of these paraphrases. The paper tackles this challenge by proposing a method to distinguish high-quality, semantically equivalent paraphrases from low-quality ones that lose crucial information.

Methods

The core of the proposed method, ParaQD, lies in its novel unsupervised data augmentation strategy. The authors designed 10 distinct operators: 4 "positive" operators that aim to create semantically equivalent paraphrases while introducing variation (e.g., backtranslation, unit expansion, number-to-word conversion) and 6 "negative" operators that intentionally perturb critical information to create unsolvable or semantically incorrect paraphrases (e.g., important phrase deletion, numerical entity deletion, unit replacement). These augmented pairs are then used to train a paraphrase scoring model using triplet loss. The model learns to map paraphrases to a latent space where high-quality paraphrases are closer to the original question than low-quality ones. For inference, cosine similarity between the encoded original question and its paraphrase is used as the score.

Datasets

The authors utilized several datasets for their experiments:

  • AquaRAT: An algebraic dataset with 30,000 training, 254 validation, and 220 testing problems.
  • EM_Math: A dataset from a partner company containing mathematics questions for grades 6-10.
  • SAWP (Simple Arithmetic Word Problems): A collected dataset of 200 algebraic problems.
  • PAWP (Paraphrased Algebraic Word Problems): A dataset of 400 algebraic word problems manually created by academicians.

For testing, they generated synthetic test sets for AquaRAT, EM_Math, and SAWP using a separate set of operators to evaluate generalization. PAWP was used for zero-shot evaluation of human-crafted paraphrases.

Key Insights and Ideas

  • The critical nature of information in AWPs: Unlike general sentence paraphrasing, AWPs are highly sensitive to the loss or alteration of specific elements like numbers and units, which can completely change the problem's solvability and meaning.
  • Targeted negative augmentation is key: The authors' success hinges on carefully designed negative augmentation operators that mimic common failure modes of paraphrasers, effectively teaching the model to identify flawed paraphrases.
  • Self-supervised learning is viable without human labels: The proposed method demonstrates that high-quality paraphrase scoring can be achieved without relying on large, manually annotated datasets for AWPs, which are scarce.
  • ParaQD's ability to cluster negatives: Interestingly, the embedding plots show that ParaQD not only separates positives from negatives but also clusters the negatives together, suggesting a deeper understanding of what constitutes a "bad" paraphrase.

Key Results

ParaQD significantly outperforms existing state-of-the-art self-supervised methods.

  • Performance Boost: It achieves up to a 32% improvement in weighted F1 score on the AquaRAT dataset compared to the best baseline.
  • Strong Separation: ParaQD achieves a separation metric (µ⁺ - µ⁻) of 0.78 on AquaRAT, compared to 0.137 for the best baseline, indicating a much clearer distinction between valid and invalid paraphrases.
  • Zero-Shot Performance: The method demonstrates impressive zero-shot performance on datasets like SAWP and PAWP, indicating its ability to generalize to unseen data distributions and even human-crafted paraphrases.
  • Robustness: The performance is shown to be robust across different encoders and seed values.

Limitations and Future Work

  • Numerical Quantity Preservation: The paper acknowledges that ParaQD might not always assign high scores to paraphrases with valid numerical changes that preserve the solution. This is a limitation, and the authors suggest future work could involve integrating an automatic word problem solver to handle such cases.
  • Operator Importance: While the combination of operators is effective, the paper notes that operator importance can vary depending on the dataset's characteristics.
  • Future Directions: The authors plan to use the ParaQD scoring model as an objective to guide language models for paraphrasing AWPs and explore its application in solvable problem detection.

Summary updated

Abstract

This paper introduces the novel task of scoring paraphrases for Algebraic Word Problems (AWP) and presents a self-supervised method for doing so. In the current online pedagogical setting, paraphrasing these problems is helpful for academicians to generate multiple syntactically diverse questions for assessments. It also helps induce variation to ensure that the student has understood the problem instead of just memorizing it or using unfair means to solve it. The current state-of-the-art paraphrase generation models often cannot effectively paraphrase word problems, losing a critical piece of information (such as numbers or units) which renders the question unsolvable. There is a need for paraphrase scoring methods in the context of AWP to enable the training of good paraphrasers. Thus, we propose ParaQD, a self-supervised paraphrase quality detection method using novel data augmentations that can learn latent representations to separate a high-quality paraphrase of an algebraic question from a poor one by a wide margin. Through extensive experimentation, we demonstrate that our method outperforms existing state-of-the-art self-supervised methods by up to 32% while also demonstrating impressive zero-shot performance.
BibTeX
@inproceedings{gupta2022john,
  author = {Rishabh Gupta and Venktesh V and Mukesh K. Mohania and Vikram Goyal},
  title = {{'John ate 5 apples' != 'John ate some apples': Self-Supervised Paraphrase Quality Detection for Algebraic Word Problems}},
  booktitle = {ECML/PKDD},
  year = {2022},
  url = {https://openreview.net/forum?id=PWssOEkGiP},
}
Figure