Noisy Neural Language Modeling for Typing Prediction in BCI Communication

Rui Dong, David Smith, Shiran Dudy, Steven BedrickNAACL 2019

TL;DR

This paper tackles the challenge of noisy character predictions in Brain-Computer Interface (BCI) typing systems. They demonstrate that training neural language models on synthetically noisy data significantly improves their robustness and performance on real-world noisy BCI inputs, outperforming traditional n-gram models.

What Problem Does This Paper Address?

Brain-computer interfaces (BCIs) offer a lifeline for individuals with severe motor impairments, enabling them to communicate by typing. These systems typically work by translating brain signals (like EEG) into character predictions. However, the accuracy of these predictions is often compromised by noise, leading to errors in the input history. Traditional language models, which are crucial for improving typing speed and accuracy by predicting the next likely character, perform poorly when trained on clean text but then applied to noisy BCI output. This paper addresses the critical gap of how to make these language models more resilient to the inherent noise in BCI communication, ensuring they remain effective even when the input is imperfect.

Methods

The core of this paper's approach is training neural language models, specifically Long Short-Term Memory (LSTM) networks, on synthetically generated noisy data. They achieve this by randomly corrupting characters in both training and test datasets, replacing them with other characters (excluding the original correct one) at varying noise rates (1% to 40%). This "data noising" technique is applied to mimic the errors encountered in BCI systems. They also explore incorporating multiple candidate character predictions (instead of just the top one) as input to the LSTM, a strategy to handle the ambiguity arising from noisy BCI signals.

Datasets

The authors evaluate their models on two datasets:

  • New York Times (NYT) corpus: A large corpus of formal language with longer sentences and richer vocabulary.
  • SUBTLEXus corpus: A corpus that tends towards colloquial language and shorter sentences.

These corpora are split into training and testing sets, with various noise rates applied to the test sets to simulate BCI input conditions.

Key Insights and Ideas

  • Noise is a feature, not a bug (for training): Training neural language models on noisy data, rather than clean data, makes them surprisingly more robust and performant when dealing with noisy BCI inputs. This is a counter-intuitive but crucial finding.
  • LSTMs shine with noise: Recurrent neural networks (LSTMs) demonstrate a significant advantage over traditional n-gram models when trained on noisy data, maintaining better performance even at high noise rates.
  • Matching noise rates helps: Models perform best when the noise rate during training closely matches the noise rate of the test data. However, even a small amount of noise in training (e.g., 1%) can offer substantial benefits on noisy test data.
  • Errors have a localized impact: While errors do affect predictions, their impact on both MRR (Mean Reciprocal Rank) and Recall diminishes significantly with distance from the error. The effect on MRR is more pronounced and longer-lasting than on Recall.
  • Ambiguous inputs can help: Providing the model with multiple candidate character predictions (and their associated probabilities) can improve performance, especially when the additional candidates have a reasonable probability of being correct.

Key Results

  • Improved performance on noisy data: Noisy LSTM models consistently outperform their clean-trained counterparts on noisy test sets. At 30% noise, KenLM (an n-gram model) becomes no better than a unigram baseline, while LSTM_clean still maintains some performance.
  • LSTM vs. N-gram: LSTM models show more stable accuracy on noisy text compared to n-gram models. Even LSTM_clean (trained on clean text) achieves higher Recall than KenLM_noisy (trained on noisy text) on some noisy test sets.
  • Error impact: Errors affect MRR more significantly than Recall. Predictions are affected by errors up to about 40 characters away for MRR, while Recall is affected for about 5 characters.
  • Ambiguous history benefits: When incorporating ambiguous histories, LSTM performance increases by over 13% with an additional candidate having a 20% correctness probability.

Limitations and Future Work

  • Synthetic noise: The paper uses synthetically generated noise. The authors acknowledge that more realistic error distributions from actual BCI typing data would be beneficial for future experiments.
  • OCLM sensitivity: The OCLM (a joint word-character model) was found to be more sensitive to noise in ambiguous inputs, sometimes degrading performance, unlike the LSTM.
  • Future work: The authors suggest experimenting with more realistic error distributions and exploring more advanced methods for integrating ambiguous histories. They also express a desire to experiment with more realistic error distributions as more real typing data becomes available.

Summary updated

BibTeX
@inproceedings{dong2019noisy,
  author = {Rui Dong and David Smith and Shiran Dudy and Steven Bedrick},
  title = {{Noisy Neural Language Modeling for Typing Prediction in BCI Communication}},
  booktitle = {Conference of the North American Chapter of the Association for Computational Linguistics},
  year = {2019},
}
Figure