TL;DR
This paper introduces novel BiLSTM-based models for disfluency detection, incorporating residual connections, self-attention, and a "noisy training" approach. Their best model achieves state-of-the-art results on the Switchboard dataset and significantly outperforms BERT on other datasets, while being substantially smaller and more efficient.
What Problem Does This Paper Address?
Disfluencies, like "uh," "um," repetitions, and self-corrections, are common in natural speech and pose a significant challenge for downstream natural language processing tasks, especially speech translation. Accurately detecting these disfluencies is crucial for improving the intelligibility and usability of automated systems. While previous work has explored various approaches, including speech-based, parsing-based, tagging-based, and translation-based methods, this paper focuses on advancing the tagging-based approach using deep learning models, aiming to achieve high performance with greater efficiency and robustness across different datasets.
Methods
The authors propose several enhancements to BiLSTM-based models for disfluency detection, treating it as a sequence labeling problem. Key components include:
- BiLSTM with Residual Blocks: Inspired by ResNet architectures, residual connections are added to skip layers. This allows for deeper networks without performance degradation, increasing representational power.
- Self-Attention: This mechanism allows the model to weigh the importance of different words in the input sequence when computing representations, capturing long-range dependencies more effectively.
- Richer Word Representations: The models utilize pre-trained word embeddings like GloVe and ELMo, and even combine them (EGBC model), to provide more contextualized input to the BiLSTM. They explicitly chose not to use BERT embeddings, citing previous work showing BERT's performance can be matched or surpassed by fine-tuned approaches with smaller models.
- Noisy Training: To improve robustness, especially against Automatic Speech Recognition (ASR) errors, the authors introduce a noisy training strategy. During training, a small percentage of the data is augmented with artificial noise, specifically insertions, deletions, or repetitions, mimicking common ASR errors. Insertion noise was found to be the most beneficial.
- CRF Layer: A Conditional Random Field (CRF) layer is used on top of the BiLSTM to model label dependencies and improve sequence labeling accuracy.
Datasets
The primary dataset used for training and evaluation is the English Switchboard Corpus. The authors specifically use the DPS annotation level, which includes disfluency annotations, part-of-speech tags, and turn information. They follow a standard train/test split from previous work.
To evaluate the robustness of their models, they also use four additional corpora:
- CallHome: Telephone conversations between family members.
- SCOTUS: Oral arguments from the Supreme Court.
- FCIC: Hearings from the Financial Crisis Inquiry Commission.
- Interview: In-house corpus of interviews with a non-native English speaker.
These datasets are noted to have different characteristics and potential sources of noise compared to Switchboard.
Key Insights and Ideas
- Residual connections are powerful for BiLSTMs: Just as in computer vision, residual connections enable deeper, more powerful sequence models without suffering from vanishing gradients or performance degradation.
- Noisy training significantly boosts robustness: Augmenting training data with simulated ASR errors (especially insertions) makes the disfluency detection model more resilient to real-world noise and errors, leading to better performance on diverse datasets.
- Efficiency matters: The paper demonstrates that state-of-the-art performance can be achieved with models that are orders of magnitude smaller than large pre-trained models like BERT, making them more practical for deployment.
- Richer embeddings are beneficial: Combining GloVe and ELMo embeddings provides a strong foundation for the BiLSTM, significantly improving baseline performance.
Key Results
- On the Switchboard dataset, their best model (EGBC + residual + self-attention + iNoise) achieved an F1 score of 92.2, matching the performance of a fine-tuned BERT model.
- Across four non-Switchboard datasets, their proposed model (EGBC + residual + self-attention + iNoise) outperformed the BERT fine-tuned model by 1.2 F1 score on average. This highlights the superior robustness of their approach.
- The proposed models are significantly more efficient: the largest model has 2.9 million parameters, compared to BERT's 110 million parameters (a ~38x reduction).
Limitations and Future Work
- Focus on ASR errors: While the noisy training approach is effective, the paper primarily focuses on simulating ASR errors. Future work could explore other types of noise, such as those arising from multi-turn dialogue or non-native speakers.
- Further exploration of noisy training: The authors suggest diving deeper into the noisy training approach, particularly in the context of speech recognition transcriptions, to find even more optimal strategies.
- Application in speech translation: The paper expresses interest in exploring how improved disfluency detection can directly benefit speech translation applications.
- Model size: While significantly smaller than BERT, further research could explore even more compact architectures while maintaining high performance.
Summary updated