Paraphrase Generation for Semi-Supervised Learning in NLU

Eunah Cho, He Xie, William M. CampbellNAACL 2019

TL;DR

This paper introduces Para-SSL, a novel approach to improve Natural Language Understanding (NLU) systems by automatically generating paraphrased utterances. This method significantly boosts NLU performance, especially for new functionalities where labeled data is scarce, by leveraging existing paraphrasing techniques and a validation step.

What Problem Does This Paper Address?

Task-oriented dialog systems, like those powering virtual assistants, rely heavily on Natural Language Understanding (NLU) to interpret user requests. A critical challenge in developing these systems is the need for large amounts of labeled data to train robust NLU models. This is particularly problematic when introducing new functionalities, as there's often no existing user data. Traditional semi-supervised learning (SSL) methods typically rely on filtering existing user utterances from live traffic. However, this approach is not feasible for functionalities that haven't been launched yet. The paper addresses this gap by proposing a method to generate relevant training data before a functionality is live, thereby improving NLU performance without requiring actual user interactions.

Methods

The core of the proposed method, Para-SSL, involves two main stages: paraphrase generation and validation.

  1. Paraphrase Generation: The authors frame paraphrase generation as a translation task, specifically a one-to-many translation where a single utterance is transformed into multiple paraphrased versions that retain the same NLU interpretation (domain, intent, and slots). They utilize a Transformer-based neural machine translation (NMT) model, trained on both general and in-domain paraphrase corpora. To generate a diverse set of paraphrases, they employ beam search to produce an "n-best list" of candidates.
  2. Validation: To ensure the quality of generated paraphrases, a crucial validation step is implemented. Generated paraphrases are passed through the NLU model, and only those with a confidence score above a certain threshold are selected. This threshold is set to match the NLU model's reject threshold for each functionality, aiming to minimize false positives.

The paper also contrasts Para-SSL with a conventional SSL approach (referred to as "SSL") that filters live traffic utterances. They also explore a "Combined" approach that leverages both Para-SSL and SSL.

Datasets

The authors created an in-domain paraphrase corpus of approximately 1 million pairs. This corpus was built by automatically extracting paraphrases from user interactions with their dialog system, focusing on utterances that invoke the same functionality. They also used a general-domain paraphrase corpus derived from back-translated English paraphrases. For evaluation, they used NLU training data that was available before the five target functionalities were designed or launched, simulating a scenario where no related utterances exist in the paraphrase model's training data.

Key Insights and Ideas

  • Generating data is viable for pre-launch NLU improvement: The most significant insight is that generating paraphrased utterances can effectively augment training data for NLU, even before a functionality is live and user data is available.
  • Paraphrase quality matters: The validation step is critical. Simply generating many paraphrases isn't enough; filtering them based on NLU model confidence is essential to avoid introducing noisy or incorrect data.
  • Para-carrier phrases are useful: The concept of "para-carrier phrases" allows grouping utterances that share functionality but not necessarily exact entities, broadening the scope for paraphrase generation.
  • Bootstrap data is valuable throughout the lifecycle: Even in the live phase, retaining and using paraphrases generated from initial bootstrap data can provide complementary benefits to using only live annotated data.

Key Results

  • Bootstrap Phase: Para-SSL achieved significant relative Semantic Error Rate (SemER) reductions, ranging from -3.5% to -18.99% for different functionalities, compared to a baseline using only bootstrap data. This demonstrates the effectiveness of generated paraphrases when no live user data is available.
  • Live Phase:
    • Para-SSL alone achieved up to 19% relative SemER reduction without access to user utterances.
  • When combined with the conventional SSL approach (using live traffic data), Para-SSL led to up to 35% relative SemER reduction.
  • The "Combined" approach often outperformed either SSL or Para-SSL individually, suggesting synergistic benefits.
  • The paper highlights that even when Para-SSL generates fewer validated utterances than SSL, it can still achieve comparable or better results, indicating the high quality of its generated paraphrases.

Limitations and Future Work

  • Computational Cost: The paper acknowledges that the "Combined" approach, while effective, can be computationally expensive. They suggest that for production, a lighter system with comparable performance might be desirable.
  • Validation Trade-offs: The authors note that the validation step involves a trade-off between computational complexity and the diversity of validated utterances as the size of the n-best list increases. Further analysis on this trade-off is suggested.
  • Functionality Complexity: The Playlist functionality showed less improvement, which the authors attribute to its complexity (more slots and intents) leading to lower confidence scores from the NLU model, thus a lower validation rate. This suggests that more complex functionalities might require tailored approaches.
  • Future Work: The paper implicitly suggests exploring more sophisticated validation techniques and potentially adapting paraphrase generation strategies for highly complex functionalities.

Summary updated

BibTeX
@inproceedings{cho2019paraphrase,
  author = {Eunah Cho and He Xie and William M. Campbell},
  title = {{Paraphrase Generation for Semi-Supervised Learning in NLU}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure