Sentential Paraphrasing as Black-Box Machine Translation

Courtney Napoles, Chris Callison-Burch, Matt PostNAACL 2016

TL;DR

This paper introduces "language packs" that package pre-trained models for generating paraphrases of English sentences. These packs leverage the Paraphrase Database (PPDB) and the Joshua machine translation decoder, offering a simple, black-box solution for researchers to generate paraphrases without building their own systems.

What Problem Does This Paper Address?

Monolingual sentence rewriting, which includes tasks like sentence simplification, compression, and grammatical error correction, aims to produce sentences with similar meaning to an input sentence. Developing custom systems for these tasks can be resource-intensive, especially for researchers not specializing in them. This paper addresses the need for an accessible, off-the-shelf solution that allows anyone to generate sentential paraphrases easily.

Methods

The core of the solution is the "language pack," which bundles pre-trained models and necessary software. The system uses the Paraphrase Database (PPDB) 2.0, a large collection of paraphrases, to build a "monolingual translation" grammar. This grammar, along with a language model and the Joshua 6 decoder, is packaged for easy deployment on a standard Unix environment. Users can treat these packs as black boxes or customize them. The system is invoked via a single command, and an interactive web tool is provided for experimentation and parameter adjustment.

The paper highlights three pre-configured models:

  • Tuned models: Specifically for sentence compression and text simplification, optimized using metrics like PRÉCIS and SARI.
  • Hand-derived weights model: A general-purpose model where weights for features (like PPDB score and language model weight) were manually tuned by examining output across different text domains.

Datasets

  • Paraphrase Database (PPDB) 2.0: Used to derive the monolingual grammar. It contains over 200 million paraphrases extracted from 100 million sentences.
  • Gigaword v.5: A 5-gram language model is included for decoding.
  • Newswire text (WSJ 0–1), "simple" English (Britannica Elementary corpus), and general text (WaCky corpus): Used for hand-tuning the general-purpose model.
  • Parallel data: Mentioned as a resource for tuning the models, with examples including data for compression (Ganitkevitch et al., 2011) and simplification (Xu et al., 2016).

Key Insights and Ideas

  • Democratizing Paraphrasing: The most significant insight is packaging complex MT technology into an easy-to-use "black box." This lowers the barrier to entry for researchers who need paraphrasing capabilities but lack the expertise or resources to build their own systems.
  • Leveraging PPDB: The paper demonstrates the power of PPDB not just for bilingual tasks but as a rich source for monolingual sentence rewriting, effectively treating paraphrasing as a form of "monolingual machine translation."
  • Flexibility of "Black Box": While offering a simple out-of-the-box solution, the system is designed to be customizable, allowing users to tweak parameters and integrate their own language models.

Key Results

The paper presents a system that allows users to generate sentential paraphrases with a single command. It provides pre-configured models for compression, simplification, and general-purpose paraphrasing. Table 1 in the paper shows sample outputs for these models, demonstrating their ability to produce variations of input sentences. The interactive web tool allows users to experiment with different weights and see the top 10 paraphrases ranked by sentence score, with Translation Edit Rate (TER) reported for each output.

Limitations and Future Work

  • Limitations:
    • The system requires Java 7 and a Unix environment, which might be a limitation for some users.
  • The paper mentions that the hand-derived weights model was not manually evaluated, suggesting a potential for further refinement.
  • The quality of paraphrases is dependent on the PPDB score thresholds and the chosen grammar size. Larger grammars offer more diversity but potentially lower confidence.
  • Future Work:
    • The authors encourage researchers to explore modifications to the model to suit specific tasks.
  • They suggest optimizing parameters using parallel data for specific tasks, leveraging the provided pipeline scripts in the Joshua decoder.
  • The interactive tool is presented as a means to demonstrate and test models, or to hand-tune them for batch processing, implying ongoing development and refinement of the tool.

Summary updated

BibTeX
@inproceedings{napoles2016sentential,
  author = {Courtney Napoles and Chris Callison-Burch and Matt Post},
  title = {{Sentential Paraphrasing as Black-Box Machine Translation}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2016},
}
Figure