Jasper: An End-to-End Convolutional Neural Acoustic Model

Jason Li, Vitaly Lavrukhin, Boris Ginsburg, Ryan Leary, Oleksii Kuchaiev, Jonathan M. Cohen, Huyen Nguyen, Ravi Teja Gadde

Jasper (Just Another SPeech Recognizer) represents a significant milestone in Automatic Speech Recognition (ASR) by demonstrating that a deep, "all-convolutional" architecture can achieve state-of-the-art results without the complexity of recurrent neural networks (RNNs). By leveraging a modular design and a novel optimizer, Jasper bridges the performance gap between end-to-end (E2E) systems and traditional hybrid models on large-scale benchmarks like LibriSpeech.

The Challenge of End-to-End ASR

Traditional ASR systems are often fragmented, requiring separate acoustic models, pronunciation lexicons, and language models integrated via Hidden Markov Models (HMMs). While E2E systems simplify this pipeline by mapping audio features directly to text, they historically struggled to match the accuracy of hybrid systems. The authors of Jasper sought to create a model that is both computationally efficient—optimized for modern GPU hardware—and deep enough to capture the complex temporal dependencies of human speech.

Architecture and Methods

The Jasper architecture is defined by its modular BxR structure, where $B$ represents the number of blocks and $R$ represents the number of sub-blocks within each block. This design allows for easy scaling of the model's depth.

The BxR Block Structure

Each sub-block in the Jasper architecture performs a fixed sequence of four operations:

  1. 1D Convolution: Captures local temporal patterns.
  2. Batch Normalization: Stabilizes training and improves convergence.
  3. ReLU Activation: Provides non-linearity.
  4. Dropout: Acts as a regularizer to prevent overfitting.
The Jasper BxR architecture, showing the modular block structure and sub-block operations.

To facilitate the training of very deep networks (up to 54 convolutional layers), Jasper utilizes residual connections. In addition to standard residuals, the authors introduced Dense Residual (DR) connections. In a DR configuration, the output of a block is added to the inputs of all subsequent blocks, significantly improving gradient flow and enabling the training of massive models with over 333 million parameters.

Optimization with NovoGrad

A core contribution of the paper is the introduction of NovoGrad, a memory-efficient variant of the Adam optimizer. While Adam tracks second moments for every individual weight, NovoGrad computes them per layer. This reduction in the memory footprint allows for larger batch sizes and improves numerical stability, which was critical for training the largest Jasper variants on massive datasets.

Key Insights and Ablations

The development of Jasper revealed several counter-intuitive findings regarding model complexity and normalization:

  • Simplicity at Scale: While complex gated units like Gated Linear Units (GLUs) or Gated Activation Units (GAUs) showed promise in smaller models, the authors found that as the model grew deeper (e.g., Jasper 10x4), simple ReLU and Batch Normalization consistently outperformed them and were easier to optimize.
  • The Necessity of Residuals: For models deeper than 15 layers, residual connections were found to be strictly necessary; without them, the training process failed to converge.
  • Normalization Sensitivity: The authors experimented with Weight Normalization but found it led to unstable, exploding activations. Batch Normalization remained the most robust choice for this architecture.
Comparison of normalization and activation functions across different Jasper configurations, showing the superiority of Batch Norm and ReLU in larger models.

Results and Performance

Jasper achieved record-breaking results on the LibriSpeech benchmark, which consists of 960 hours of read English audiobooks. When combined with a Transformer-XL language model, Jasper reached a 2.95% Word Error Rate (WER) on the "test-clean" set. Even using simple greedy decoding (without an external language model), the model achieved a highly competitive 3.86% WER.

Beyond accuracy, Jasper's reliance on 1D convolutions makes it exceptionally fast. Unlike RNNs, which are difficult to parallelize due to their sequential nature, Jasper's convolutional layers can be processed in parallel across GPU cores, making it highly efficient for both high-throughput training and low-latency inference.

Conclusion and Future Work

Jasper proves that simple, highly optimized convolutional building blocks can outperform more complex architectures in speech recognition. While it excels at "read" speech, the authors noted that conversational speech (such as the Hub5’00 dataset) remains a challenge, with WERs rising to 16.2% on the CallHome subset. Future research will likely focus on more robust data augmentation and architectural refinements to handle the noise and overlap inherent in spontaneous human conversation.

Summary updated

BibTeX
@inproceedings{li2019jasper,
  author = {Jason Li and Vitaly Lavrukhin and Boris Ginsburg and Ryan Leary and Oleksii Kuchaiev and Jonathan M. Cohen and Huyen Nguyen and Ravi Teja Gadde},
  title = {{Jasper: An End-to-End Convolutional Neural Acoustic Model}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure