Abstract
Mask-Net is a novel architecture designed to strip away unwanted biases—such as speaker accents—from speech data using a technique called "adversarial forgetting." By incorporating global context and reducing architectural redundancy, Mask-Net achieves superior generalization in Speech-to-Text (STT) tasks. Experimental results demonstrate that Mask-Net improves Word Error Rate (WER) by up to 2.2% on out-of-distribution data compared to traditional adversarial methods.
Introduction
Deep learning models are often "shortcut learners." In Speech-to-Text (STT) tasks, a model might inadvertently rely on a speaker's accent or background noise to make predictions rather than focusing purely on linguistic content. This leads to poor performance when the model encounters a new accent it hasn't heard before.
While previous research has attempted to "forget" these unwanted features using Gradient Reversal Layers (GRL) or masking, these methods often suffer from two flaws:
- Redundancy: They often re-learn low-level features from scratch in a separate "forgetting" module, wasting training time.
- Lack of Context: They often treat each time step in a sequence independently. In speech, an accent is a global characteristic; if a model decides to "forget" an accent at the start of a sentence but "remembers" it halfway through, the resulting features become inconsistent and noisy.
The Mask-Net Architecture
To address these issues, we propose Mask-Net (M-Net), which builds on the concept of an adversarial mask. The architecture consists of four main components: an Encoder ($E_x$) that extracts features $Z$, a Forget Net ($F$) that generates a mask $M$, a Predictor ($P_x$) for the main STT task, and a Discriminator ($D_x$) that attempts to identify the unwanted bias (e.g., accent).
Key innovations in the Mask-Net design include:
- Bypassing Redundancy: Instead of feeding raw input into the Forget Net, the model uses the output of the Encoder ($Z$). This ensures the model doesn't waste resources re-learning basic audio patterns.
- Global Context Masking: Inspired by "Squeeze and Excitation" networks, the Forget Net pools information across all time steps to create a global view. It then "tiles" this mask across the entire sequence, ensuring the "forgetting" filter is applied consistently across the whole utterance.
- Isolated Adversarial Training: Gradients from the Discriminator only flow back to the Forget Net, not the main Encoder. This keeps the Encoder "pure" and driven only by the main task, preserving its utility for other downstream applications.
Experimental Results
We evaluated Mask-Net using a 34-hour subset of the Common Voice V1 dataset, focusing on STT as the main task and accent classification as the adversarial task. We compared Mask-Net against a strong Jasper-based baseline, multi-task learning approaches, and standard Gradient Reversal (GRL) techniques.
Key Findings
- Better Generalization: Mask-Net showed an absolute improvement in Word Error Rate (WER) of 2.2% on out-of-distribution test sets and 1.3% on in-distribution sets compared to traditional models.
- Superior Invariance: When tested on its ability to hide accents, Mask-Net outperformed the standard GRL technique by an absolute 5% lead in achieving truly invariant features.
- Temporal Consistency: By "tiling" the mask, the model ensures that embeddings map to a similar feature space throughout the audio clip, preventing the "jitter" seen in previous masking attempts.
Conclusion and Future Work
Mask-Net demonstrates that global context and feature reuse are critical for effective adversarial forgetting in speech processing. While the "squeeze and excitation" method provides effective global context, future work will explore attention-based forget net modules to identify specific signal components for masking. Additionally, we plan to scale experiments to larger datasets with a wider variety of biases beyond accents.
Summary updated