Jasper (Just Another SPeech Recognizer) represents a significant milestone in Automatic Speech Recognition (ASR) by demonstrating that a deep, "all-convolutional" architecture can achieve state-of-the-art results without the complexity of recurrent neural networks (RNNs). By leveraging a modular design and a novel optimizer, Jasper bridges the performance gap between end-to-end (E2E) systems and traditional hybrid models on large-scale benchmarks like LibriSpeech.
The Challenge of End-to-End ASR
Traditional ASR systems are often fragmented, requiring separate acoustic models, pronunciation lexicons, and language models integrated via Hidden Markov Models (HMMs). While E2E systems simplify this pipeline by mapping audio features directly to text, they historically struggled to match the accuracy of hybrid systems. The authors of Jasper sought to create a model that is both computationally efficient—optimized for modern GPU hardware—and deep enough to capture the complex temporal dependencies of human speech.
Architecture and Methods
The Jasper architecture is defined by its modular BxR structure, where $B$ represents the number of blocks and $R$ represents the number of sub-blocks within each block. This design allows for easy scaling of the model's depth.
The BxR Block Structure
Each sub-block in the Jasper architecture performs a fixed sequence of four operations:
- 1D Convolution: Captures local temporal patterns.
- Batch Normalization: Stabilizes training and improves convergence.
- ReLU Activation: Provides non-linearity.
- Dropout: Acts as a regularizer to prevent overfitting.
To facilitate the training of very deep networks (up to 54 convolutional layers), Jasper utilizes residual connections. In addition to standard residuals, the authors introduced Dense Residual (DR) connections. In a DR configuration, the output of a block is added to the inputs of all subsequent blocks, significantly improving gradient flow and enabling the training of massive models with over 333 million parameters.
Optimization with NovoGrad
A core contribution of the paper is the introduction of NovoGrad, a memory-efficient variant of the Adam optimizer. While Adam tracks second moments for every individual weight, NovoGrad computes them per layer. This reduction in the memory footprint allows for larger batch sizes and improves numerical stability, which was critical for training the largest Jasper variants on massive datasets.
Key Insights and Ablations
The development of Jasper revealed several counter-intuitive findings regarding model complexity and normalization:
- Simplicity at Scale: While complex gated units like Gated Linear Units (GLUs) or Gated Activation Units (GAUs) showed promise in smaller models, the authors found that as the model grew deeper (e.g., Jasper 10x4), simple ReLU and Batch Normalization consistently outperformed them and were easier to optimize.
- The Necessity of Residuals: For models deeper than 15 layers, residual connections were found to be strictly necessary; without them, the training process failed to converge.
- Normalization Sensitivity: The authors experimented with Weight Normalization but found it led to unstable, exploding activations. Batch Normalization remained the most robust choice for this architecture.
Results and Performance
Jasper achieved record-breaking results on the LibriSpeech benchmark, which consists of 960 hours of read English audiobooks. When combined with a Transformer-XL language model, Jasper reached a 2.95% Word Error Rate (WER) on the "test-clean" set. Even using simple greedy decoding (without an external language model), the model achieved a highly competitive 3.86% WER.
Beyond accuracy, Jasper's reliance on 1D convolutions makes it exceptionally fast. Unlike RNNs, which are difficult to parallelize due to their sequential nature, Jasper's convolutional layers can be processed in parallel across GPU cores, making it highly efficient for both high-throughput training and low-latency inference.
Conclusion and Future Work
Jasper proves that simple, highly optimized convolutional building blocks can outperform more complex architectures in speech recognition. While it excels at "read" speech, the authors noted that conversational speech (such as the Hub5’00 dataset) remains a challenge, with WERs rising to 16.2% on the CallHome subset. Future research will likely focus on more robust data augmentation and architectural refinements to handle the noise and overlap inherent in spontaneous human conversation.
Summary updated