Abstract
End-to-end (E2E) speech recognition models have revolutionized the field but remain opaque "black boxes," particularly regarding their failure to handle diverse accents. This paper investigates the internal logic of DeepSpeech 2 to localize and quantify how accent information is encoded. By applying gradient-based attribution, mutual information measures, and probing classifiers, the study reveals that accent information peaks in the first recurrent layer (RNN0). Furthermore, it demonstrates that while information mixing is an architectural property, non-native accents suffer from significant phonetic misalignment and higher uncertainty in deeper layers. These findings pinpoint the first recurrent layer as a primary target for interventions to improve accent robustness.
Introduction
Modern Automatic Speech Recognition (ASR) has transitioned from modular systems to end-to-end (E2E) architectures that map audio directly to text. While more efficient, these models are notoriously sensitive to variations in speech, particularly accents. A model trained primarily on standard US English often fails when encountering Indian or Scottish accents. This research aims to move beyond observing these failures to understanding where in the neural network the confusion occurs and how accent-specific information propagates through the layers.
Methodology
The study utilizes the DeepSpeech 2 (DS2) architecture—comprising two convolutional layers followed by five bidirectional LSTM (RNN) layers—and employs three analytical techniques:
- Gradient-based Attribution: Sensitivity analysis of output to input frames, using Earth Mover’s Distance (EMD) to quantify the misalignment between model focus and actual phonetic timing.
- Information-Theoretic Measures: Computation of Mutual Information (MI) between hidden representations and labels (accent and phone) to track information flow.
- Probing Classifiers: Training auxiliary classifiers on frozen layers to determine the discriminative power of each layer for accent and phone prediction.
Datasets
The research primarily uses the Mozilla Common Voice dataset, specifically a curated subset where the same sentences are spoken across seven English accents (African, Australian, Canadian, England, Indian, Scotland, and US) to control for linguistic content. Additionally, TIMIT serves as a clean benchmark for phone-level analysis, and the model is pre-trained on the 960-hour LibriSpeech corpus.
Key Insights and Results
The RNN0 Bottleneck and Accent Encoding
Probing classifiers and Mutual Information analysis consistently identified the first recurrent layer (RNN0) as the peak for accent information. An accent classifier reached its highest accuracy (~33%) at this layer. Beyond RNN0, accent information begins to dissipate as the model shifts focus toward mapping audio to abstract text. This suggests that the initial transition from convolutional features to recurrent temporal modeling is where the network most strongly encodes speaker-specific accent traits.
Alignment Sensitivity and Phonetic Confusion
Native accents (US and Canadian) exhibited tight alignment between the model's gradients and phonetic boundaries. In contrast, for heavily accented speech such as Indian or Scottish, the model's "attention" was physically smeared across the timeline, resulting in higher EMD scores. This misalignment is a key driver of the performance gap; for instance, the model achieved an 18.7% Word Error Rate (WER) on Canadian speech but struggled at 49.1% WER for Indian accents.
Information Mixing as an Architectural Property
The study found that "Information Mixing"—the process where the model's focus on a specific phone blurs into surrounding context—is largely a property of the architecture. As shown in the layer-wise analysis, the decrease in phone focus is nearly identical across all accents, indicating that the network's fundamental way of processing context does not change, even when the input data is challenging.
Conclusion and Future Work
The localization of accent information to the RNN0 layer provides a clear roadmap for future architectural improvements. The authors suggest that this layer is the ideal site for adversarial training. By forcing the representations in the first recurrent layer to be accent-invariant through an adversarial loss, it may be possible to build ASR systems that are significantly more robust to global variations in speech. Future research should explore whether these findings hold for Transformer-based architectures and non-English languages.
Summary updated