Abstract
Traditional automatic speech recognition (ASR) systems often use an acoustic model (AM) built on handcrafted acoustic features, such as log Mel-filter bank (FBANK) values. Recent studies found that AMs with convolutional neural networks (CNNs) can directly use the raw waveform signal as input. Given sufficient training data, these AMs can yield a competitive word error rate (WER) to those built on FBANK features. This paper proposes a novel multi-span structure for acoustic modelling based on the raw waveform with multiple streams of CNN input layers, each processing a different span of the raw waveform signal. Evaluation on both the single channel CHiME4 and AMI data sets show that multi-span AMs give a lower WER than FBANK AMs by an average of about 5% (relative). Analysis of the trained multi-span model reveals that the CNNs can learn filters that are rather different to the log Mel-filters. Furthermore, the paper shows that a widely used single span raw waveform AM can be improved by using a smaller CNN kernel size and increased stride to yield improved WERs.
Introduction
Traditional Automatic Speech Recognition (ASR) systems rely heavily on handcrafted features like Mel-filter banks (FBANK) or Mel-frequency cepstral coefficients (MFCCs). These features are designed based on human auditory perception models, which may inadvertently discard acoustic information that is useful for machine learning models. While Convolutional Neural Networks (CNNs) have shown promise in learning directly from raw waveforms, most existing models utilize a single fixed temporal window (span). This paper addresses the limitation of single-span models, which may fail to capture the diverse time-frequency resolutions inherent in speech signals, by proposing a multi-span architecture.
Methodology
The proposed Multi-Span Acoustic Model architecture utilizes parallel streams of CNN layers to process raw audio.
Key components of the methodology include:
- Parallel Streams: The raw audio is fed into multiple parallel streams, each consisting of two consecutive CNN layers.
- Variable Spans: By fixing the output map size and kernel size while varying the stride, each stream effectively processes a different length (span) of the raw waveform. This allows the model to capture features at multiple temporal resolutions simultaneously.
- Optimized Kernels: The research demonstrates that smaller CNN kernels (e.g., 50 samples) are significantly more effective than the larger kernels (e.g., 400 samples) used in earlier waveform models.
- Minimal Constraints: The initial layers avoid common techniques like log-compression or max-pooling to allow the model to learn the most effective representations directly from the data.
- Integration: The outputs from all streams are concatenated and processed through a standard Deep Neural Network (DNN) for final classification.
Experimental Results and Analysis
The model was evaluated on the CHiME4 (noisy far-field speech) and AMI (conversational speech) datasets.
Performance Comparison
On the CHiME4 dataset, the multi-span model achieved a 17.1% Word Error Rate (WER), outperforming both the FBANK baseline (18.1%) and a standard single-span waveform model (20.2%). Similarly, on the AMI dataset, the multi-span approach achieved a 29.3% WER, compared to 31.1% for the FBANK baseline. Overall, the multi-span architecture provided a consistent ~5% relative improvement over traditional handcrafted features.
Filter Analysis
Analysis of the trained filters indicates that the model learns representations that are distinct from traditional Mel-filters.
The model automatically learns "wavelet-like" filters. Even when provided with larger kernels, the network often learns to zero out weights at the edges, effectively optimizing its own kernel size for the best resolution. This departure from Mel-filter behavior suggests the model captures acoustic nuances that handcrafted features typically miss.
Conclusion
The multi-span acoustic model demonstrates that learning directly from raw waveforms using multiple temporal contexts can significantly improve ASR performance. By utilizing smaller kernels and parallel streams, the model captures complementary information that outperforms traditional FBANK features. Future work may involve scaling this front-end with larger architectures like Transformers and exploring diverse non-linearities to further enhance feature diversity.
Summary updated