TL;DR
This paper integrates a simplified, physiologically plausible model of the human cochlea (as a trainable filterbank) into a large, state-of-the-art pre-trained acoustic model (wav2vec2) for speech recognition. The authors demonstrate that this hybrid approach can be trained effectively using self-supervision and fine-tuning, yielding interpretable filter structures and achieving performance comparable to standard models, while avoiding artifacts seen in previous, less sophisticated models.
What Problem Does This Paper Address?
The field of automatic speech recognition (ASR) has largely moved towards "black-box" end-to-end deep learning models, often trained on massive datasets. While these models achieve impressive performance, they offer little insight into the underlying physiological processes of human hearing. Conversely, traditional auditory research aims to understand these processes, but often struggles to leverage the power of modern large-scale neural networks and datasets. This paper bridges this gap by exploring how to integrate a biologically inspired component (a cochlear model) into a powerful pre-trained ASR system, aiming to gain insights into both auditory processing and the learned representations within these large models. Specifically, they revisit a previous finding where a simpler model learned unexpected "wide-band" filter structures, and investigate whether this behavior persists in a more powerful, pre-trained context.
Methods
The core of this work is the integration of a SincNet filterbank, a trainable convolutional filter designed to mimic aspects of the cochlea, into the wav2vec2 architecture. Instead of using wav2vec2's standard convolutional neural network (CNN) encoder, the researchers replace it with the SincNet filters. The training process involves a two-stage approach:
- Pre-training (Self-Supervision): The model is initially trained using the self-supervised contrastive loss objective of wav2vec2. Crucially, the learning rate schedulers are decoupled, allowing the SincNet encoder to train while the rest of the network (transformer layers) is frozen. This prevents catastrophic forgetting of the pre-trained knowledge.
- Fine-tuning: After the initial self-supervised training, the entire model is fine-tuned on labeled data.
The researchers experiment with various configurations, including removing a max-pooling layer (which they previously found to be physiologically relevant), adjusting kernel sizes, and varying the number of filters. They also explore incorporating the SincNet filters directly into the self-supervision phase.
Datasets
The primary dataset used for training and evaluation is LibriSpeech. The paper details the different subsets used for various experiments, including:
dev-cleananddev-otherfor evaluation and validation.train-100,train-360, andtrain-500for self-supervision and fine-tuning.
The paper does not introduce any new benchmarks.
Key Insights and Ideas
- Physiological Plausibility in Black Boxes: It's possible to integrate physiologically inspired components (like SincNet filters) into powerful, pre-trained ASR models without sacrificing performance. This opens doors for ASR research to also contribute to understanding auditory processing.
- Avoiding Artifacts with Scale: The unexpected "wide-band" filter structures observed in their previous work with simpler models were largely absent when using the large, pre-trained wav2vec2 context network. This suggests that the powerful pre-trained layers might already encode the information that the simpler model was trying to learn with broad filters.
- Decoupled Learning Rates are Key: Using decoupled learning rate schedulers is essential for effectively training specific parts of a pre-trained model (like the SincNet encoder) without degrading the performance of the frozen components.
- SincNet as a Viable Alternative: SincNet, when integrated into a pre-trained framework, performs comparably to standard CNN encoders and can converge faster when trained from scratch.
- Interpretability Gains: While SincNet doesn't necessarily outperform a generic CNN in terms of ASR accuracy in this context, it offers more interpretable filter structures that align better with known speech features like formants.
Key Results
- The hybrid SincNet-wav2vec2 model achieved Word Error Rates (WER) below 4% on the LibriSpeech dataset, comparable to the baseline wav2vec2 model.
- Removing the max-pooling layer after the SincNet filters improved performance, aligning with previous findings on physiological relevance.
- Longer training durations (100k updates) led to further performance improvements.
- When trained from scratch, SincNet encoders converged faster than CNN encoders.
- The emergence of "wide-band" filters, a previously observed artifact, was significantly reduced or eliminated in the hybrid model, especially when using a limited number of filters (around 40). Increasing the number of filters did not lead to the reappearance of these wide-band structures.
Limitations and Future Work
- Simplicity of the Cochlear Model: The SincNet model used is a simplified representation of the cochlea. The authors acknowledge that more complex physiological models, incorporating active systems, are a goal for future research.
- Pre-trained Context Inhibits Wide-Band Filters: The experiments suggest that the pre-trained transformer layers in wav2vec2 might already encode the information that led to wide-band filters in simpler models, thus preventing their emergence in the trainable filter layer. This implies that the learned representations in large models might be different from what simpler models learn.
- Computational Cost: Training large self-supervised models, especially with complex filterbanks, is computationally intensive. The paper notes that the pre-train-fine-tune approach requires more data and time than pure fine-tuning.
- Future Work: The authors suggest that this approach can be extended to more intricate physiological models to gain deeper insights into how biological mechanisms interpret external inputs. They also highlight the potential for this framework to facilitate a better understanding of the evolution of physiological mechanisms in interpreting sensory data.
Summary updated