LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses

Ebrahim Feghhi, Junlin Hu, Nima Hadidi, Jonathan C. Kao

Abstract

A promising pathway for restoring communication in patients with dysarthria and anarthria is speech neuroprostheses, which directly decode speech from cortical neural activity. Two benchmarks, Brain-to-Text '24 and '25, released intracranial recordings from patients with dysarthria along with a baseline algorithm trained with Connectionist Temporal Classification (CTC). Despite significant innovation on these benchmarks, all leading published prior work relies on a WFST-based CTC decoder that requires ~320 GB of RAM. These memory requirements limit accessibility for both patients and researchers. Here, we propose LightBeam, a non-WFST based CTC decoder that requires only ~10 GB of RAM and achieves state-of-the-art performance on both benchmarks. LightBeam achieves this by integrating an LLM into the beam-search process via delayed fusion, obviating the prior need for using a large N-gram LM. LightBeam is implemented in Python and is open-source.

Introduction

Speech neuroprostheses are life-changing devices that decode cortical neural activity directly into text, allowing patients with severe speech impairments to communicate. However, the current "gold standard" for decoding—Connectionist Temporal Classification (CTC) paired with a Weighted Finite-State Transducer (WFST)—is extremely resource-intensive. Leading algorithms require approximately 320 GB of RAM, primarily due to their reliance on massive 5-gram language models. This memory bottleneck restricts deployment to high-end server clusters, hindering privacy-preserving local use and broader research accessibility.

This study utilizes two major intracranial recording benchmarks: Brain-to-Text ’24 (B2T ’24), involving 12,100 sentences from a participant with ALS, and Brain-to-Text ’25 (B2T ’25), involving 10,948 sentences from a second participant. We introduce LightBeam, a decoder designed to achieve state-of-the-art accuracy while running on standard consumer-grade hardware.

Methods

LightBeam replaces the rigid WFST framework with a flexible, GPU-accelerated approach. Its architecture is defined by three key innovations:

  1. Delayed Fusion with LLMs: Traditionally, Large Language Models (LLMs) are used only for final rescoring. LightBeam integrates a small LLM (Llama 3.2 1B) directly into the first-pass beam search. By updating hypothesis scores at specified intervals (e.g., every 5–20 encoder frames), the decoder can prune unlikely paths much earlier and more accurately than N-gram models.
  2. Vectorized Lexicon: To eliminate CPU-GPU synchronization bottlenecks, the traditional trie-based lexicon is replaced with a dense transition table. This allows the system to check thousands of word extensions in parallel on the GPU.
  3. Hybrid Language Modeling: By combining a small 4-gram model for word-boundary expansion with the LLM for long-range context, LightBeam eliminates the need for massive 320 GB N-gram models.
The rescore interval allows for a tunable trade-off between decoding speed (RTF) and accuracy (WER).

Results

LightBeam achieves a massive reduction in resource requirements while improving decoding accuracy. As shown in the performance comparison below, RAM usage dropped from ~320 GB to ~10 GB (a 32x reduction), while VRAM usage remained compatible with mid-tier consumer GPUs (~6 GB).

Comparison of LightBeam and WFST-based decoders across performance and resource metrics.

Key findings include:

  • State-of-the-Art Accuracy: LightBeam outperformed the baseline WFST decoder on both benchmarks. When combined with a final Generative Error Correction (GEC) step using Llama 3.1 8B, it achieved a new SOTA Word Error Rate (WER) of 5.01% on B2T '24 and 2.08% on B2T '25.
  • Real-Time Performance: The Real-Time Factor (RTF) remained significantly below 1.0 (averaging 0.06 to 0.14), meaning the system decodes speech much faster than it is produced.
  • Tunable Efficiency: The "delayed fusion" mechanism allows researchers to balance latency and accuracy. Increasing the rescore interval significantly lowers the computational load with minimal impact on error rates.

Conclusion

LightBeam demonstrates that integrating LLMs into the beam-search process via delayed fusion can replace massive N-gram models, making high-performance speech neuroprostheses accessible on consumer-grade hardware. While the system currently requires a fine-tuning phase to adapt to a user's specific vocabulary, future work will explore continuous adaptation and testing in complex, long-form conversational settings.

Summary updated

Abstract

A promising pathway for restoring communication in patients with dysarthria and anarthria is speech neuroprostheses, which directly decode speech from cortical neural activity. Two benchmarks, Brain-to-Text '24 and '25, released intracranial recordings from patients with dysarthria along with a baseline algorithm trained with Connectionist Temporal Classification (CTC). Despite significant innovation on these benchmarks, all leading published prior work relies on a WFST-based CTC decoder that requires ${\sim}$320 GB of RAM. These memory requirements limit accessibility for both patients and researchers. Here, we propose LightBeam, a non-WFST based CTC decoder that requires only ${\sim}$10 GB of RAM and achieves state-of-the-art performance on both benchmarks. LightBeam achieves this by integrating an LLM into the beam-search process via delayed fusion, obviating the prior need for using a large N-gram LM. LightBeam is implemented in Python and is open-source.
BibTeX
@misc{feghhi2026lightbeam,
  author = {Ebrahim Feghhi and Junlin Hu and Nima Hadidi and Jonathan C. Kao},
  title = {{LightBeam: An Accurate and Memory-Efficient CTC Decoder for Speech Neuroprostheses}},
  year = {2026},
  eprint = {2603.14002},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2603.14002},
}
Figure