TL;DR
This paper tackles the challenge of making powerful, non-incremental Transformer models (like BERT) work for real-time disfluency detection in conversational systems. They achieve this by adapting training and decoding strategies, enabling these models to process speech word-by-word with minimal latency while retaining high accuracy, a crucial step for responsive human-computer interaction.
What Problem Does This Paper Address?
Conversational systems, particularly those used in mental health care, are increasingly relying on understanding user speech. Disfluencies (like "uh," "um," and speech repairs) are important indicators of cognitive states and can signal difficulties in processing semantic information, as seen in patients with Alzheimer's Disease. While traditional disfluency detection methods often work offline on complete transcripts, they are unsuitable for live, interactive systems that require immediate processing as speech arrives. Existing incremental models, though faster, often sacrifice accuracy. The core problem is bridging this gap: how to leverage the high accuracy of state-of-the-art non-incremental models (like Transformers) for real-time, incremental disfluency detection without significant performance degradation.
Methods
The authors adapt Transformer models, specifically BERT, for incremental disfluency detection through modifications in both training and decoding.
- Incrementalizing BERT: They modify the training procedure using "chunk-based" training (breaking sentences into smaller chunks) and "add-M" training (progressively adding more data). For real-time processing, they employ "prophecy-based" decoding, where a language model (GPT-2) predicts future word sequences to provide the necessary rightward context that BERT models inherently lack.
- Decoding Strategies: Three common decoding methods (Beam search, Top-k sampling, and Top-p sampling) are used to generate these future word predictions.
- Input Features: Beyond word embeddings, they experiment with Part-of-Speech (POS) tags and word timings (duration from the end of the previous word to the current one) as additional features.
- Utterance Segmentation: A separate word-by-word utterance segmentation system is used to process incoming ASR results before feeding them to the disfluency detector.
Datasets
The primary dataset used is the Switchboard corpus, a standard for conversational speech research. They also use a subset of the Penn Treebank III release for validation and testing. For ASR hypotheses, they utilize data generated by IBM's Watson Speech-To-Text service.
Key Insights and Ideas
- Bridging the Gap: The core insight is that by carefully adapting training and using predictive language models for "prophecies" of future speech, non-incremental models like BERT can be made to work effectively in an incremental setting.
- Prophecy-Based Decoding: Using GPT-2 to predict future word sequences is a key enabler, providing the necessary context for BERT to make accurate predictions incrementally.
- Add-M Training: The "add-M" training strategy, particularly with N=1 and M=1, proves beneficial for improving both final output accuracy and incremental performance.
- Top-p Sampling: This decoding strategy emerges as the most effective for generating future word predictions, balancing accuracy and stability.
- Trade-off Management: The paper highlights the trade-off between incremental performance (latency, stability) and final output accuracy, demonstrating that their approach can achieve a good balance.
Key Results
- State-of-the-Art Performance: Their best performing incremental BERT model achieves state-of-the-art results on disfluency detection for both pre-segmented transcripts and raw ASR results.
- Repair Onset Detection: Achieved an $F_{rps}$ score of 0.853 on pre-segmented transcripts and 0.802 on automatically segmented ASR results.
- Edit Term Detection: Achieved high scores of 0.960 and 0.944 on pre-segmented and unsegmented settings, respectively.
- Incremental Performance: The system demonstrates a fast repair detection latency, with the first detection of a repair onset occurring on average within 0.31 words. Edit overhead (a measure of output stability) is also competitive.
- Add-M Training Advantage: The add-M training regime consistently shows improvements in both final output and incremental performance across different models.
Limitations and Future Work
- Longer Repairs: The system still struggles with detecting longer repairs compared to explicit backtracking mechanisms. The strictly left-to-right decoding constraint imposes memory limitations.
- Future Work: The authors suggest exploring more efficient ways to navigate the decoding space for longer repairs. They also plan to use this system to help detect cognitive conditions like Alzheimer's Disease in live diagnostic systems.
Summary updated