Abstract
In this work, we present the DSSformer, an evolution of the Conformer architecture that replaces standard depthwise temporal convolutions with Diagonal State Space (DSS) models. DSS layers represent a variant of linear Recurrent Neural Networks (RNNs) derived from discretizing linear dynamical systems with diagonal state transition matrices. By projecting input sequences onto a space of orthogonal polynomials, the DSSformer achieves superior modeling of temporal dynamics. We evaluate our approach on three major benchmarks: Switchboard (300h and 2000h) and the MALACH archive of Holocaust survivor testimonials. Our results demonstrate state-of-the-art performance, achieving 8.9% and 6.7% WER on Switchboard 300/2000h respectively, and a 7% relative improvement on MALACH. Furthermore, we provide an empirical analysis showing that DSS layers effectively learn damped Fourier basis functions with layer-specific attenuation and consistent frequency spacing.
Introduction
For several years, the Conformer architecture has served as the backbone of Automatic Speech Recognition (ASR). Its success stems from the integration of self-attention, which captures global context, and depthwise convolutions, which handle local patterns. However, depthwise convolutions are constrained by fixed-size kernels and lack a rigorous theoretical framework for approximating complex temporal signals.
Recent advancements in Structured State Space models (S4) have introduced a powerful alternative for sequence modeling by treating data as a continuous linear dynamical system. In this paper, we investigate the Diagonal State Space (DSS) model—a simplified and efficient version of S4—as a replacement for the convolution module within the Conformer block. The resulting architecture, the DSSformer, offers both improved accuracy and greater mathematical interpretability.
The DSSformer Architecture
The core innovation of the DSSformer is the substitution of the Conformer's temporal convolution module with a DSS layer.
Linear Dynamical Systems and Diagonalization
The DSS layer models an input sequence by discretizing a continuous state-space model. While original S4 models utilized complex "diagonal plus low-rank" (DPLR) matrices to maintain mathematical properties like the HiPPO (High-Order Polynomial Projection Operators) initialization, the DSS approach utilizes a strictly diagonal transition matrix. This simplification streamlines implementation without sacrificing the model's ability to project sequences onto orthogonal polynomial spaces.
Computational Efficiency
One of the primary advantages of the DSS layer is its efficiency. By applying the Circular Convolution Theorem, the DSS layer can be computed as a full-sequence convolution in $O(T \log T)$ time using Fast Fourier Transforms (FFTs). This provides a significant scaling advantage over the $O(T^2)$ complexity of standard self-attention when processing long audio sequences.
Experimental Setup
We validated the DSSformer across three diverse and challenging datasets:
- Switchboard (300 hours): The standard benchmark for conversational English telephone speech.
- Switchboard + Fisher (2000 hours): A large-scale version of the telephone speech task to test scalability.
- MALACH (176 hours): A corpus of Holocaust survivor testimonials characterized by heavy accents, age-related speech variations, and high emotional intensity.
In all experiments, we compared the DSSformer against a baseline Conformer using a Neural Transducer framework.
Results and Analysis
The DSSformer consistently outperformed the baseline Conformer across all benchmarks:
- Switchboard 300h: Achieved a Word Error Rate (WER) of 8.9% on the Hub5 '00 combined set, compared to 11.2% for the baseline.
- Switchboard 2000h: Reached 6.7% WER, establishing a new state-of-the-art for single non-attention-encoder-decoder (non-AED) models.
- MALACH: Achieved a 20.2% WER (when combined with a Transformer-XL language model), representing a 7% relative improvement over previous best-published results.
Interpretability: Learning Damped Fourier Functions
A key contribution of this research is the analysis of the learned DSS transition matrices. We found that the layers converge to damped Fourier basis functions. Specifically:
- Frequency Consistency: The imaginary parts of the eigenvalues (representing frequency) converge to nearly identical, linearly-spaced values across all layers.
- Layer-Specific Attenuation: The real parts of the eigenvalues (representing the decay or "forgetting" rate) are layer-specific. This suggests that different layers learn to smooth or retain historical context at varying temporal scales.
Conclusion
The DSSformer demonstrates that Diagonal State Space models are a highly effective replacement for temporal convolutions in speech recognition. By leveraging the strengths of linear dynamical systems, the DSSformer achieves superior accuracy on conversational and accented speech while providing a more interpretable mechanism for temporal modeling. Future work will focus on optimizing the parameter count—currently 73M for DSSformer versus 63M for Conformer—and exploring the potential of keeping transition matrices fixed during training to further improve robustness.
Summary updated