Paraphrastic Language Models

Xunying Liu, Mark J. F. Gales, Philip C. WoodlandInterspeech 2012 · OpenReview

Introduction

Standard n-gram language models (LMs) serve as the backbone for most modern speech recognition systems. However, these models are inherently "surface-level" thinkers; they calculate the probability of a word based strictly on the immediate preceding word sequence. This approach struggles with the inherent flexibility of natural language, where the same underlying meaning can be expressed through numerous variations (e.g., "I really like" versus "I generally prefer").

When a specific word sequence is absent or infrequent in training data, standard LMs assign it a low probability, regardless of how common the underlying meaning might be. This limitation, known as poor context coverage, has historically been addressed using word-level synonyms from manually curated databases like WordNet. However, such methods fail to capture longer phrase-level relationships and do not scale efficiently across new languages or domains.

Methodology

The researchers introduce "Paraphrastic Language Models" (PLMs), a framework that treats a sentence as one of many possible surface forms of an underlying semantic meaning. The core of the PLM is the maximization of the marginal probability across all possible paraphrase variants rather than a single word sequence.

To implement this, the authors utilized Weighted Finite State Transducers (WFSTs) to represent the complex mappings between words, phrase segmentations, and paraphrase variants. This allows for the generation of "paraphrase lattices"—compact representations of multiple semantically similar sentences—which can be used to train the model efficiently.

WFST representation of (a) phrase mappings and (b) weighted paraphrase variants used for training.

Key components of the methodology include:

  1. Automatic Phrase Extraction: Instead of relying on manual labels, the system uses distributional similarity. Phrases that frequently appear within the same left and right contexts (e.g., "a couple of" and "several") are automatically flagged as potential paraphrases.
  2. Multi-level Modeling: The researchers combined word-level and phrase-level LMs using log-linear interpolation. This hybrid approach captures both fine-grained word patterns and broader semantic relationships while mitigating "modeling confusion"—a state where the model becomes less certain about specific word choices due to increased coverage.

Datasets and Evaluation

The approach was evaluated on a Large Vocabulary Continuous Speech Recognition (LVCSR) task using approximately 1.0 billion words from eight sources. Primary data included the Fisher conversational telephone speech transcriptions (20M words) and the UWWeb conversational web data (525M words).

Summary of paraphrase sources, extraction methods, and the resulting scale of phrase pairs.

The study compared automatically extracted pairs from conversational data against expert-labeled pairs from WordNet. Evaluation was performed on the dev04 set, which consists of three hours of Fisher conversational telephone speech.

Key Results and Insights

The paraphrastic approach consistently outperformed baseline models across different scales of training data:

  • Fisher-only Training: The multi-level PLM reduced the Word Error Rate (WER) from 17.5% to 17.0%, a significant 0.5% absolute improvement over a non-paraphrastic multi-level baseline.
  • Large-scale Training: When trained on the full Fisher and UWWeb datasets, the PLM achieved a WER of 16.2%, outperforming the standard word-level 4-gram baseline of 16.7%.
  • Data Efficiency: Remarkably, a PLM trained on 545 million words outperformed a standard model trained on twice as much data (1 billion words) by 0.2% absolute.

One of the most significant insights was that data-driven extraction outperformed expert-derived databases. Adding paraphrases from WordNet provided only marginal gains compared to the millions of pairs extracted automatically from conversational data. This suggests that the statistical patterns found in actual speech are more relevant for recognition tasks than formal semantic hierarchies.

Limitations and Future Work

While effective, the automatic extraction method occasionally identifies antonyms (e.g., "I love" vs. "I hate") as paraphrases because they share similar contexts. While this increases context coverage, it could lead to semantic inaccuracies in certain applications. Additionally, training on paraphrase lattices increases computational complexity compared to standard n-gram training.

Future research directions include exploring "directed paraphrasing" to adapt models to specific styles or domains and developing more sophisticated extraction techniques to ensure the grammatical integrity of generated variants.

Summary updated

BibTeX
@inproceedings{liu2012paraphrastic,
  author = {Xunying Liu and Mark J. F. Gales and Philip C. Woodland},
  title = {{Paraphrastic Language Models}},
  booktitle = {Interspeech},
  year = {2012},
  url = {https://openreview.net/forum?id=XDt3NTG9NQR},
}
Figure