Planning and Generating Natural and Diverse Disfluent Texts as Augmentation for Disfluency Detection

Jingfeng Yang, Diyi Yang, Zhaoran MaEMNLP 2020 · OpenReview

Abstract: Jingfeng Yang, Diyi Yang, Zhaoran Ma. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP).

2020.

Introduction

Spontaneous speech is rarely perfect. It is filled with stutters, repetitions, and self-corrections known as disfluencies. For AI systems like voice assistants, these disfluencies act as noise that must be filtered to understand intent. While the Switchboard (SWBD) corpus is the standard for training detection models, its limited size necessitates data augmentation. Traditional heuristic methods—like random word repetition—often fail to capture the complexity of human speech, particularly "substitutions" where a speaker changes their mind mid-sentence.

The Planner-Generator Framework

To address the limitations of simple heuristics, the authors propose a two-stage Planner-Generator (PG) framework designed to create realistic synthetic disfluencies.

The architecture of the Planner-Generator (PG) framework for generating disfluent text.
  1. The Planner: This component identifies where a disfluency should occur within a fluent sentence. It treats the task as sequence tagging, marking potential interruption points.
  2. The Generator: This component determines what the disfluent segment (the reparandum) should be. Using an encoder-decoder architecture, it generates text that logically leads into the "repair" segment of the original sentence.

The researchers discovered that decoupling these stages allows for better control. Interestingly, a "noisier" generator variant (PG-NC-AD-ID) produced the most diverse disfluencies, which proved more effective for training robust detection models than a more "accurate" but conservative generator.

Data Augmentation and Scale

The PG model was trained on Switchboard and then applied to millions of fluent sentences from the WMT2017 Monolingual News Crawl. The team evaluated the impact of adding 3 million and 20 million synthetic sentences to the training pipeline, testing how scale influences model performance.

Key Results

The proposed approach achieved new state-of-the-art results in disfluency detection. By augmenting the training data with 3M synthetic sentences, the BERT-based model reached an F-score of 92.2, which increased to 92.3 with 20M sentences.

Performance comparison of the proposed BERT-based models against existing baselines on the Switchboard corpus.

Beyond quantitative metrics, human evaluations via Amazon Mechanical Turk confirmed the "naturalness" of the generated text. The PG model scored 0.76 on naturalness, significantly outperforming the 0.65 achieved by traditional heuristic methods. The model was particularly effective at reducing false negatives in complex substitution and deletion errors.

Insights and Future Work

A major takeaway is that diversity trumps accuracy in data augmentation. A generator that produces a wide variety of errors—even rare ones—helps the final model generalize better. However, the model still relies on an initial set of human-annotated data to learn disfluency patterns. Future research could explore applying these techniques to low-resource languages or using natural disfluencies to improve the human-like quality of Text-to-Speech (TTS) systems.

Summary updated

BibTeX
@inproceedings{yang2020planning,
  author = {Jingfeng Yang and Diyi Yang and Zhaoran Ma},
  title = {{Planning and Generating Natural and Diverse Disfluent Texts as Augmentation for Disfluency Detection}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2020},
  url = {https://openreview.net/forum?id=xRE5GIDRkxI},
}
Figure