FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities

Lilit Grigoryan, Vladimir Bataev, Nikolay Karpov, Andrei Andrusenko, Vitaly Lavrukhin, Boris GinsburgarXiv 2025 · DOI · arXiv

Introduction

In modern Automatic Speech Recognition (ASR), the computational burden has shifted. While neural network encoders have seen massive acceleration through GPU optimization, the decoding stage—specifically Connectionist Temporal Classification (CTC) beam search—remains a significant bottleneck. Traditional decoders often rely on sequential CPU processing or rigid, difficult-to-extend C++ implementations. FlexCTC addresses this by moving the entire decoding process, including language model (LM) fusion and phrase boosting (PB), onto the GPU within a Python-native framework.

Bridging the Research-to-Production Gap

FlexCTC is designed to provide both the flexibility required for research and the high-throughput performance needed for production. Unlike existing solutions that often sacrifice extensibility for speed, FlexCTC maintains a PyTorch-centric workflow that allows for rapid iteration without compromising on hardware acceleration.

Feature comparison of FlexCTC against other industry-standard decoders.

As shown in the comparison above, FlexCTC is unique in offering a fully open-source, Python-native implementation that supports both CPU and GPU execution while integrating advanced features like language modeling and phrase boosting.

Technical Innovations

The performance gains in FlexCTC stem from several key architectural choices:

  • Two-Level Parallelism: The system parallelizes both across batches of audio files and across multiple hypotheses within each beam, maximizing GPU utilization.
  • GPU-Native Components: By implementing a trie-based hypothesis manager and an n-gram language model (NGPU-LM) directly on the GPU, FlexCTC eliminates the latency associated with CPU-GPU data transfers.
  • CUDA Graphs: To mitigate the overhead of launching numerous small GPU kernels, FlexCTC uses CUDA Graphs to encapsulate the entire decoding loop into a single, highly efficient executable.
  • Contextual Boosting: Phrase-level boosting for "hotwords" (e.g., specialized medical or financial terms) is integrated directly into the pipeline using a prefix tree, allowing for domain-specific accuracy improvements with minimal runtime overhead.

Performance Evaluation

The researchers evaluated FlexCTC on three diverse datasets: SPGI (financial), Earnings21 (financial benchmark), and MultiMed (medical). To test contextualization in specialized domains, LLaMA 3.3-70B was used to automatically extract a list of ~1,000 medical terms for phrase boosting.

Performance comparison (WER and RTFx) of FlexCTC vs. state-of-the-art decoders across multiple datasets.

Key Results

  • Throughput: FlexCTC consistently achieves a 2–3x improvement in Real-Time Factor (RTFx) compared to highly optimized C++ decoders like Flashlight and PyCTCDecode. At larger beam sizes (e.g., beam=16), the performance gap widens further due to superior GPU scaling.
  • Accuracy: The integration of LM fusion and phrase boosting resulted in a 10.5% relative reduction in Word Error Rate (WER) on medical data and an 8.5% reduction on financial data.
  • Efficiency: At a batch size of 128, the performance gap between simple "greedy" decoding and complex beam search (size 4) was narrowed to just 10.3%, making high-quality decoding viable for high-throughput environments.

Conclusion

FlexCTC demonstrates that a Python-native, GPU-accelerated approach can outperform traditional C++ decoders while maintaining the ease of use necessary for rapid iteration. By eliminating the CPU bottleneck and integrating contextual features directly into the GPU pipeline, it provides a robust solution for high-performance ASR in specialized domains. Future work will look toward integrating neural language models and extending these techniques to Transducer-based architectures.

Summary updated

Abstract

While beam search improves speech recognition quality over greedy decoding, standard implementations are slow, often sequential, and CPU-bound. To fully leverage modern hardware capabilities, we present a novel open-source FlexCTC toolkit for fully GPU-based beam decoding, designed for Connectionist Temporal Classification (CTC) models. Developed entirely in Python and PyTorch, it offers a fast, user-friendly, and extensible alternative to traditional C++, CUDA, or WFST-based decoders. The toolkit features a high-performance, fully batched GPU implementation with eliminated CPU-GPU synchronization and minimized kernel launch overhead via CUDA Graphs. It also supports advanced contextualization techniques, including GPU-powered N-gram language model fusion and phrase-level boosting. These features enable accurate and efficient decoding, making them suitable for both research and production use.
BibTeX
@misc{grigoryan2025flexctc,
  author = {Lilit Grigoryan and Vladimir Bataev and Nikolay Karpov and Andrei Andrusenko and Vitaly Lavrukhin and Boris Ginsburg},
  title = {{FlexCTC: GPU-powered CTC Beam Decoding With Advanced Contextual Abilities}},
  year = {2025},
  eprint = {2508.07315},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2508.07315},
}
Figure