Abstract
Developing robust conversational AI requires a vast array of training utterances to map diverse user inputs to specific knowledge base intents. This paper presents an automated pipeline that leverages an ensemble of paraphrasing techniques—including backtranslation and lexical substitution—coupled with a sophisticated selection algorithm using BERT and Universal Sentence Encoder (USE). Our approach significantly increases the quantity and quality of generated paraphrases, outperforming single-model generative approaches and reducing the manual burden of dataset curation.
Introduction
A primary challenge in building effective chatbots and virtual assistants is linguistic variability. Users express the same intent in myriad ways (e.g., "How do I pay?" vs. "I want to settle my bill"). Traditionally, developers manually curate these "utterances," a process that is time-consuming, costly, and often fails to capture natural language diversity. While automated paraphrase generation exists, it frequently suffers from low diversity or semantic "hallucinations." This research introduces a system designed to bridge this gap by automatically generating a large pool of candidate utterances and intelligently filtering them for semantic accuracy and lexical variety.
Methodology: The "Overgenerate and Select" Pipeline
The proposed system follows a three-stage "overgenerate and select" paradigm designed to maximize both the breadth and precision of the output.
- Summarization: For lengthy knowledge base articles, the system first performs extractive summarization to isolate the core information.
- Ensemble Paraphrase Generation: To ensure linguistic variety, the system employs four distinct techniques:
- Full Backtranslation (BT): Translating English to German and back to English to find natural variations.
- Phrase-level Backtranslation (NP/VP): Isolating and translating specific noun or verb phrases to preserve structure while varying vocabulary.
- Synonym Replacement (WN): Utilizing WordNet synsets for word-level swaps.
- Phrase Replacement (PPDB): Leveraging the Paraphrase Database for multi-word expressions.
- Candidate Selection: The "secret sauce" of the pipeline involves a multi-stage filter. It uses Universal Sentence Encoder (USE) for initial similarity checks and a BERT-based classifier to ensure strict semantic equivalence. Finally, deduplication algorithms based on USE similarity and word-count diversity ensure the final set is not repetitive.
Evaluation and Results
The system was evaluated against public benchmarks (STS, MRPC) and a private dataset of 567 real-world sentences from technical manuals and FAQs.
As shown in the evaluation metrics, the full ensemble (BT + NP + VP + WN + PPDB) using BERT filtering achieved an average of 5.97 useful paraphrases per input sentence, a nearly five-fold increase over standard backtranslation (1.16).
Furthermore, the inclusion of the BERT-based filter was critical. While USE provides a good baseline for similarity, BERT was far more effective at identifying "adversarial" examples where word overlap is high but meaning has shifted. This improved the "fraction of useful" paraphrases to 0.44 in the ensemble. Notably, purely generative models like Conditional Variational Autoencoders (CVAE) performed poorly in these specialized domains, yielding a useful fraction of only 0.05.
Conclusion
This research highlights that an ensemble of structured paraphrasing techniques, when combined with deep-learning-based filtering, provides a robust solution for automated utterance generation. By prioritizing both semantic accuracy and lexical diversity, the pipeline offers a scalable alternative to manual curation. Future work will explore fine-tuning large language models to further refine the generation stage and evaluate the direct impact of these automated utterances on the end-to-end performance of live conversational agents.
Summary updated