Deformable TDNN with adaptive receptive fields for speech recognition

Keyu An, Yi Zhang, Zhijian Ou

Abstract

Time Delay Neural Networks (TDNNs) are widely used in both DNN-HMM based hybrid speech recognition systems and recent end-to-end systems. Nevertheless, the receptive fields of TDNNs are limited and fixed, which is not desirable for tasks like speech recognition, where the temporal dynamics of speech are varied and affected by many factors. This paper proposes to use deformable TDNNs for adaptive temporal dynamics modeling in end-to-end speech recognition. Inspired by deformable ConvNets, deformable TDNNs augment the temporal sampling locations with additional offsets and learn the offsets automatically based on the ASR criterion, without additional supervision. Experiments show that deformable TDNNs obtain state-of-the-art results on WSJ benchmarks (1.42%/3.45% WER on WSJ eval92/dev93 respectively), outperforming standard TDNNs significantly. Furthermore, we propose the latency control mechanism for deformable TDNNs, which enables deformable TDNNs to do streaming ASR without accuracy degradation.

Introduction: The Limitation of Fixed Receptive Fields

In modern Automatic Speech Recognition (ASR), Time Delay Neural Networks (TDNNs)—essentially 1D convolutional layers—serve as efficient building blocks due to their speed and parallelizability. However, standard TDNNs suffer from a fundamental constraint: their receptive fields are fixed.

A standard TDNN samples the input signal at a rigid, pre-defined grid (e.g., $t-2, t, t+2$). This fixed temporal window is applied uniformly across all speech segments, regardless of whether a speaker is talking quickly, slowly, or emphasizing specific phonemes. While "multi-stream" architectures attempt to address this by manually designing multiple grids of varying sizes, these solutions are often cumbersome and increase parameter counts. Ideally, a model should dynamically adapt its temporal "view" based on the acoustic content.

Comparison of fixed (left) and deformable (right) receptive fields in a TDNN.

Proposed Method: Deformable TDNNs

To address the rigidity of standard temporal filters, this paper introduces the Deformable TDNN, inspired by deformable convolutional networks in computer vision. The core idea is to allow the network to "stretch" or "shift" its sampling points to better capture the varying rhythms of human speech.

Architecture and Offset Prediction

The Deformable TDNN augments the standard sampling grid with learnable offsets. As shown in the architecture below, a small auxiliary 1D convolutional layer is added to predict these offsets ($\Delta t$) for each sampling point based on the input features.

Architecture of the Deformable TDNN layer showing offset prediction and irregular sampling.
  1. Learnable Offsets: Instead of a fixed interval, the sampling locations become $t + \Delta t$.
  2. Linear Interpolation: Since predicted offsets are often fractional, the model employs linear interpolation to compute the signal value at non-integer time steps.
  3. Latency Control: To ensure the model remains viable for real-time streaming, the authors implement a clip mechanism. This penalty during training prevents the model from looking too far into the future, maintaining a manageable latency window without significant accuracy loss.

The entire architecture is optimized within a CTC-CRF (Connectionist Temporal Classification - Conditional Random Field) framework, which enhances data efficiency compared to standard CTC.

Experimental Setup and Results

The model was evaluated on the Wall Street Journal (WSJ) dataset, comprising approximately 80 hours of training data. Performance was measured on the standard dev93 and eval92 test sets. To ensure robustness, the authors applied 3-fold speed perturbation and SpecAugment (time-warping) during training.

Key Findings

  • State-of-the-Art Performance: Combined with an RNN language model, the system achieved a Word Error Rate (WER) of 1.42% on WSJ eval92 and 3.45% on dev93, setting a new benchmark for this dataset.
  • Parameter Efficiency: The transition from standard to deformable layers added negligible overhead (e.g., increasing parameters from 11.90M to 11.93M) while providing a relative error reduction of nearly 10%.
  • Robustness to Temporal Warping: Deformable TDNNs are notably more resilient to artificial time-warping in test data, maintaining performance where fixed-grid models degrade.
  • Streaming Capability: With latency control, the model achieved a 2.60% WER on eval92 in a streaming setup, far surpassing a standard TDNN clipped at test time (4.38% WER).

Conclusion

Deformable TDNNs provide a lightweight yet powerful mechanism for adaptive temporal modeling in ASR. By allowing the model to learn its own receptive fields, it achieves superior accuracy and robustness with minimal parameter increase. Future work will focus on scaling this approach to larger datasets like LibriSpeech and integrating deformable layers into hybrid architectures with self-attention or LSTM components.

Summary updated

Abstract

Time Delay Neural Networks (TDNNs) are widely used in both DNN-HMM based hybrid speech recognition systems and recent end-to-end systems. Nevertheless, the receptive fields of TDNNs are limited and fixed, which is not desirable for tasks like speech recognition, where the temporal dynamics of speech are varied and affected by many factors. This paper proposes to use deformable TDNNs for adaptive temporal dynamics modeling in end-to-end speech recognition. Inspired by deformable ConvNets, deformable TDNNs augment the temporal sampling locations with additional offsets and learn the offsets automatically based on the ASR criterion, without additional supervision. Experiments show that deformable TDNNs obtain state-of-the-art results on WSJ benchmarks (1.42\%/3.45\% WER on WSJ eval92/dev93 respectively), outperforming standard TDNNs significantly. Furthermore, we propose the latency control mechanism for deformable TDNNs, which enables deformable TDNNs to do streaming ASR without accuracy degradation.
BibTeX
@misc{an2021deformable,
  author = {Keyu An and Yi Zhang and Zhijian Ou},
  title = {{Deformable TDNN with adaptive receptive fields for speech recognition}},
  year = {2021},
  eprint = {2104.14791},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2104.14791},
}
Figure