Adaptive Unsupervised Self-training for Disfluency Detection

Zhongyuan Wang, Yixuan Wang, Shaolei Wang, Wanxiang CheCOLING 2022 · OpenReview

Abstract: Supervised methods have achieved remarkable results in disfluency detection. However, in real-world scenarios, human-annotated data is difficult to obtain. Recent works try to handle disfluency detection with unsupervised self-training, which can exploit existing large-scale unlabeled data efficiently. However, their self-training-based methods suffer from the problems of selection bias and error accumulation. To tackle these problems, we propose an adaptive unsupervised self-training method for disfluency detection. Specifically, we re-weight the importance of each training example according to its grammatical feature and prediction confidence. Experiments on the Switchboard dataset show that our method improves 2.3 points over the current SOTA unsupervised method. Moreover, our method is competitive with the SOTA supervised method.

Introduction

Spontaneous speech is often characterized by disfluencies—interruptions like "um," "uh," or mid-sentence corrections (e.g., "I want a cup of tea um I mean coffee"). Detecting these disfluencies is essential for cleaning transcripts for downstream NLP tasks such as translation or summarization. While supervised models excel at this task, they require vast amounts of expensive human-annotated data.

Unsupervised self-training has emerged as a promising alternative, where a "teacher" model labels unlabeled data for a "student" model to learn from. However, traditional self-training faces two major hurdles: selection bias, where the model only learns from samples it is already confident about, and error accumulation, where mistakes made by the teacher are propagated to the student. To address these, researchers from the Harbin Institute of Technology have developed an adaptive unsupervised self-training method that re-weights training examples based on grammatical features and prediction confidence.

Methods

The proposed framework moves beyond simple filtering of labels. Instead, it employs an adaptive re-weighting strategy for every training example based on two primary factors:

  1. Contrastive Grammar Checking: Unlike previous models that used simple grammar checks, this method introduces a contrastive approach. It compares the sentence's grammaticality before and after removing suspected disfluencies. If the "cleaned" version is significantly more grammatical, the model's confidence in the detection increases.
  2. Label Confidence with Temperature: To prevent the model from over-committing to incorrect labels in early training stages, a temperature mechanism is used to dampen high-confidence scores.
The Adaptive Unsupervised Self-training framework involves iterative cycles of pseudo-data construction, teacher labeling, re-weighting, and student training.

The workflow begins by training a weak teacher on synthetic "pseudo-data" generated by simple rules. This teacher then labels real, unlabeled ASR (Automatic Speech Recognition) data. These labels are re-weighted using the contrastive grammar checker and confidence scores before being used to train a student model. In the next iteration, the student becomes the new teacher, and the cycle repeats.

Key Results

The method was evaluated primarily on the English Switchboard dataset, the gold standard for disfluency detection. Despite not using the human-annotated training set, the model achieved an F1 score of 90.3, outperforming the previous unsupervised state-of-the-art (SOTA) by 2.3 points.

Performance comparison of supervised and unsupervised methods on the Switchboard dataset. Our method significantly outperforms previous unsupervised benchmarks and rivals some supervised models.

Notably, the unsupervised model's performance (90.3 F1) is now competitive with BERT-based models trained on 60,000 human-labeled sentences (90.5 F1). Robustness was also demonstrated through cross-domain evaluations on CallHome, SCOTUS, and FCIC datasets, where the model consistently outperformed previous unsupervised baselines. Ablation studies confirmed that both the grammar checker and confidence re-weighting are vital for performance.

Conclusion

This adaptive re-weighting approach effectively mitigates selection bias and error accumulation in unsupervised self-training. While the initial teacher still relies on rule-based pseudo-data and the iterative process is computationally intensive, the results demonstrate that unsupervised methods can rival supervised performance in disfluency detection. Future work may explore applying this re-weighting strategy to other low-resource NLP tasks where human labels are scarce but unlabeled data is plentiful.

Summary updated

Abstract

Supervised methods have achieved remarkable results in disfluency detection. However, in real-world scenarios, human-annotated data is difficult to obtain. Recent works try to handle disfluency detection with unsupervised self-training, which can exploit existing large-scale unlabeled data efficiently. However, their self-training-based methods suffer from the problems of selection bias and error accumulation. To tackle these problems, we propose an adaptive unsupervised self-training method for disfluency detection. Specifically, we re-weight the importance of each training example according to its grammatical feature and prediction confidence. Experiments on the Switchboard dataset show that our method improves 2.3 points over the current SOTA unsupervised method. Moreover, our method is competitive with the SOTA supervised method.
BibTeX
@inproceedings{wang2022adaptive,
  author = {Zhongyuan Wang and Yixuan Wang and Shaolei Wang and Wanxiang Che},
  title = {{Adaptive Unsupervised Self-training for Disfluency Detection}},
  booktitle = {International Conference on Computational Linguistics},
  year = {2022},
  url = {https://openreview.net/forum?id=0B4TROMUrM},
}
Figure