TL;DR
Researchers from IBM have introduced the DSSformer, a new architecture for speech recognition that replaces the standard temporal convolutions in Conformers with Diagonal State Space (DSS) models. This swap leads to state-of-the-art performance on major benchmarks like Switchboard and MALACH, while providing a more mathematically interpretable way to model speech dynamics.
What Problem Does This Paper Address?
For the past few years, the Conformer has been the "gold standard" architecture for Automatic Speech Recognition (ASR). It works by combining the global reach of Transformers (self-attention) with the local pattern-matching of depthwise convolutions.
However, depthwise convolutions have limitations: they use fixed-size kernels and lack a strong theoretical grounding in how they approximate signals over time. Recently, Structured State Space models (S4) emerged as a powerful alternative for modeling long sequences by treating them as linear dynamical systems. This paper investigates whether a simplified version—Diagonal State Space (DSS)—can replace the convolution module in Conformers to create a more efficient and accurate speech recognizer.
Methods
The authors propose the DSSformer, which modifies the standard Conformer block by swapping the depthwise convolution layer for a DSS layer.
Key technical components include:
- Linear Dynamical Systems: The DSS layer models the input sequence as a linear Recurrent Neural Network (RNN) obtained by discretizing a continuous state-space model.
- Diagonalization: Unlike the original S4 models which use complex "diagonal plus low-rank" matrices, DSS uses a strictly diagonal transition matrix. This is simpler to implement and, as the authors show, loses no performance.
- Efficiency: By using the Circular Convolution Theorem, the DSS layer can be computed as a full-sequence convolution in $O(T \log T)$ time using Fast Fourier Transforms (FFTs), making it much more efficient than the $O(T^2)$ complexity of standard self-attention for long sequences.
- Initialization: The authors experimented with several initialization schemes for the state transition matrix, including HiPPO (based on orthogonal polynomials) and S4D (linear and inverse scalings).
Datasets
The model was evaluated on three rigorous public corpora:
- Switchboard (300 hours): The standard benchmark for English conversational telephone speech.
- Switchboard + Fisher (2000 hours): A much larger scale version of the telephone speech task.
- MALACH (176 hours): A challenging archive of Holocaust survivor testimonials. This dataset is particularly difficult due to heavy accents, age-related speech patterns, and high emotional content.
Key Insights and Ideas
- Complementary Strengths: The authors echo previous findings that self-attention and DSS layers are not redundant. While DSS is great at capturing local temporal dynamics (like a convolution), it doesn't "subsume" the global context provided by self-attention.
- Learning Damped Fourier Functions: One of the most interesting "aha" moments is the analysis of what the DSS layers actually learn. The authors found that the layers converge to damped Fourier basis functions.
- Consistency Across Layers: Interestingly, the "frequency" components (imaginary parts of the eigenvalues) converge to almost identical linearly-spaced values across all layers. However, the "attenuation" or decay coefficients (real parts) are layer-specific, suggesting each layer learns to "forget" or "smooth" the past at different rates.
- Initialization is Key: The choice of how to start the transition matrix matters. Initializing with a simple linear imaginary part ($\lambda_n = -1 + i \cdot n$) performed as well as or better than more complex mathematical initializations.
Key Results
The DSSformer consistently outperformed the standard Conformer across all tasks:
- Switchboard (300h): Achieved a Word Error Rate (WER) of 8.9% on the Hub5 '00 combined set, a significant improvement over the 11.2% of the baseline Conformer.
- Switchboard (2000h): Reached 6.7% WER, setting a state-of-the-art result for single non-attention-encoder-decoder (non-AED) models.
- MALACH: Improved the WER by 7% relative over the previous best-published results, reaching 20.2% when combined with a Transformer-XL language model.
Limitations and Future Work
- Parameter Count: The DSSformer is slightly larger than the baseline Conformer (73M vs. 63M parameters). While the performance gain is clear, the authors suggest future work could involve combining the DSS mixing matrix with other layers to simplify the architecture and reduce overhead.
- Fixed Eigenvalues: The authors propose investigating whether the transition matrices could be kept fixed during training. This would reduce the number of trainable parameters and potentially make the model even more robust.
- Broader Application: While this paper focused on Neural Transducers, the authors plan to study the effectiveness of DSS for other end-to-end ASR frameworks.
Summary updated