Incremental Disfluency Detection for Spoken Learner English

Lucy Skidmore, Roger MooreNAACL 2022

TL;DR

This paper introduces a novel approach to detecting disfluencies in spoken English from language learners by incorporating non-lexical features like laughter and silence. By processing speech incrementally, this method aims to provide real-time feedback in educational applications, showing improved precision, especially when learner errors co-occur with disfluencies.

What Problem Does This Paper Address?

Spoken language is naturally filled with disfluencies like hesitations, repetitions, and false starts. Detecting these is crucial for understanding communicative intent, particularly in the context of Computer-Assisted Language Learning (CALL). Existing research often focuses on removing disfluencies for downstream tasks and typically processes entire utterances at once. This non-incremental approach is unsuitable for real-time dialogue systems that need to respond word-by-word. Furthermore, learner speech presents unique challenges: disfluencies are more frequent, reparandum phrases are longer, and crucially, disfluencies often co-occur with grammatical and lexical errors. This makes automatic detection in learner speech particularly difficult. The paper aims to address these limitations by developing an incremental disfluency detection system specifically for learner English, incorporating features that might better capture the nuances of non-native speech.

Methods

The core of the proposed method is an incremental disfluency detection system built upon a combination of an LSTM (Long Short-Term Memory) model and an HMM (Hidden Markov Model) decoder. This architecture processes speech incrementally, considering a window of up to nine words (the current word and eight preceding words) to detect disfluencies at the point of repair onset.

What's novel here is the incorporation of two specific non-lexical features:

  • Silence: Explicitly encoded as the presence or absence of a preceding pause, or implicitly through audio features.
  • Laughter: Treated as a "trouble management device" in learner speech, its presence or absence is also used as an input feature.

These features are derived directly from the transcripts and assigned to each word, indicating pauses, laughter, or laughed words. The system then uses these features, along with POS tags and word embeddings, as input to the LSTM, which feeds into the HMM for sequence labeling.

Datasets

The primary dataset used is the NICT-JLE Corpus. This is a transcription-only corpus of spoken English from Japanese learners, containing approximately 300 hours of speech. It includes labels for edit terms, disfluencies, and non-verbal sounds (silence and laughter). Crucially, a subset of the corpus (167 interviews) also contains labels for learners' morphological, grammatical, and lexical errors, which is vital for evaluating the impact of learner errors on disfluency detection.

Key Insights and Ideas

  • Laughter as a Signal: The paper pioneers the use of laughter as a feature for disfluency detection in learner speech. This is motivated by observations that learners use laughter to manage uncertainty and signal trouble, making it a potentially valuable paralinguistic cue.
  • Silence and Laughter Synergy: While silence and laughter individually show only minor improvements, their combination leads to a noticeable boost in performance, particularly in detecting disfluencies that co-occur with learner errors. This suggests that these non-lexical cues work together to provide a richer signal.
  • Learner Errors are a Major Hurdle: The study confirms that learner errors significantly degrade disfluency detection performance, especially when errors precede the repair phrase. This highlights the challenge of "non-typical" lexical data in learner speech.
  • Incremental Advantage for Learner Speech: The incremental approach, by focusing on repair onset, seems to be more robust to the challenges posed by learner errors compared to non-incremental methods that rely more on syntactic parallelism.

Key Results

  • When combined, silence and laughter features improve both repair start and reparandum phrase detection F-scores compared to a baseline model.
  • The combined features significantly boost precision for detecting disfluencies that co-occur with learner errors, with precision seeing a particularly notable increase.
  • The proposed incremental model significantly outperforms a previous incremental approach on learner speech (Moore et al., 2015) on the NICT-JLE Corpus, setting a new benchmark for this specific task.
  • While performance does not reach the level of state-of-the-art non-incremental approaches, the gains in incremental detection for learner speech are substantial.

Limitations and Future Work

  • Transcription-Only Data: A significant limitation is the reliance on transcribed data. The authors note that using actual audio would likely yield even better results, as silence and laughter detection from transcripts is less reliable than from audio.
  • Limited Learner Error Data: Learner error tags were only available for the test set, preventing their use as features during training. This limits the ability to jointly model learner errors and disfluencies.
  • Future Directions:
    • Multimodal Corpora: The development of multimodal corpora (audio, video, transcripts) for dialogue-based CALL is strongly advocated.
  • Advanced Models: Testing incrementalized BERT-based detectors and using POS taggers specifically trained for learner English are suggested for further improvement.
  • Other Paralinguistic Features: Exploring additional non-lexical features like gestures and gaze, which are known to signal disfluency, is a promising avenue.
  • Joint Modeling: Investigating the relationship between learner errors and disfluencies by modeling them jointly is a key future research direction.

Summary updated

BibTeX
@inproceedings{skidmore2022incremental,
  author = {Lucy Skidmore and Roger Moore},
  title = {{Incremental Disfluency Detection for Spoken Learner English}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2022},
}
Figure