Introduction
Training robust Speech-to-Text (STT) systems requires large, diverse datasets to overcome variability caused by "nuisances" like speaker accents. These biases can lead models to learn non-generalizable representations, causing performance to degrade when encountering unfamiliar speech patterns. This paper introduces mask-Net, a novel architecture that employs "adversarial forgetting" (AF) to strip away these unwanted biases while retaining essential linguistic content.
Methodology
The mask-Net architecture refines the adversarial forgetting framework to improve efficiency and temporal consistency. Unlike previous methods that treat every time step independently, mask-Net introduces a context-aware masking layer.
Architecture Components
- Encoder ($E$): Processes raw audio into an intermediate representation ($Z$).
- Forget Network ($F$): Instead of processing raw input, the forget net takes the encoder's output ($Z$). This avoids redundant feature learning and focuses the network on high-level representations.
- Squeeze and Excitation: To ensure the mask is consistent across an entire utterance, the forget net uses a "squeeze" step (global average pooling) and an "excitation" step (linear layers). This allows the model to consider the entire audio sequence before generating a mask.
- Masking and Adversarial Training: A mask ($M$) is multiplied by the encoder output to produce invariant features ($\tilde{Z}$). A discriminator ($D$) attempts to identify the accent from these features, while the forget net is trained to maximize the discriminator's error. Crucially, adversarial gradients are decoupled, flowing only back to the forget net to preserve the encoder's general-purpose strength.
Experimental Setup
The model was evaluated using a 34-hour subset of the Common Voice V1 dataset. Performance was measured on an in-distribution (ID) test set and two out-of-distribution (OOD) sets: TestIN (Indian accents) and TestNZ (New Zealand accents). The primary metrics were Word Error Rate (WER) and the accuracy of the accent discriminator.
Results and Discussion
Experimental results demonstrate that mask-Net outperforms both standard baselines and traditional Gradient Reversal Layer (GRL) methods in both transcription accuracy and bias removal.
As shown in the results table, mask-Net achieved an absolute WER improvement of 1.3% on in-distribution tests and an average of 2.2% on out-of-distribution sets compared to the baseline. Furthermore, the discriminator's ability to guess the speaker's accent dropped to 41% (compared to 46.7% for GRL), indicating that the model was more successful at stripping away accent information. The use of global context through tiling the mask ensured that embeddings remained stable throughout the utterance.
Conclusion
mask-Net provides a more efficient and context-aware approach to inducing invariance in speech models. By leveraging the encoder's existing features and enforcing temporal consistency, the model achieves better generalization on diverse accents. Future work will explore scaling this approach to larger datasets and implementing attention-based mechanisms for more sophisticated masking.
Summary updated