Abstract
This paper introduces a novel approach to low-resource Spoken Command Recognition (SCR) by combining Adversarial Reprogramming (AR) with Transfer Learning (TL). By applying trainable perturbations to input audio and using a similarity-based label mapping technique, the authors successfully repurpose a pretrained English acoustic model to recognize commands in Lithuanian, Arabic, and dysarthric Mandarin using only a handful of training samples. The proposed AR+TL system demonstrates remarkable data efficiency, outperforming existing state-of-the-art methods in extreme low-data regimes.
Introduction
Spoken Command Recognition (SCR) is the technology behind voice-activated smart home devices and digital assistants. Modern deep learning SCR systems are highly accurate but incredibly data-hungry, requiring massive datasets of labeled audio to train effectively. This creates a significant barrier for "low-resource" scenarios, such as developing voice interfaces for rare languages or for users with speech impairments like dysarthria. While standard Transfer Learning (TL) or massive self-supervised models (like wav2vec 2.0) can help, they often require substantial computational hardware or still need a decent chunk of target data to fine-tune properly. This study tackles the challenge of building a highly accurate SCR system with as few as 3 to 20 audio samples per word.
Methodology
The researchers propose an AR-SCR system that relies on three core techniques to adapt a source model to a target domain.
Adversarial Reprogramming (AR)
Instead of updating the weights of a neural network, AR adds a trainable "noise" (perturbation) directly to the input audio signal. This effectively translates the target domain audio (e.g., Lithuanian speech) into a format that a frozen, pretrained source model (trained on English) can understand.
Similarity-Based Label Mapping
Because the pretrained model outputs English word probabilities, the system needs a way to map these to the target language. Instead of mapping them randomly, the authors use cosine similarity to map target classes to source classes that share similar acoustic characteristics in the model's latent space. For example, the Lithuanian word "ne" was automatically mapped to the English words "nine" and "no".
Combining AR with Transfer Learning (AR+TL)
The ultimate system uses AR as a front-end processor to modify the input signals, and TL as a back-end processor to fine-tune the acoustic model's weights simultaneously. To further prevent overfitting, SpecAugment is integrated to generate more training data dynamically.
Experimental Results
The proposed AR+TL system was evaluated on three low-resource datasets: Lithuanian, Dysarthric Mandarin, and Arabic.
Performance and Data Efficiency
The system demonstrated exceptional accuracy across all tasks. On the Lithuanian dataset with just 3 training samples per class, AR+TL achieved 58.9% accuracy (compared to a 33.8% baseline). With 20 samples, it reached 88.6%, outperforming a state-of-the-art wav2vec-based model in comparative tests. In the Dysarthric Mandarin task, the system achieved 82.3% accuracy, a 28.59% relative improvement over the baseline. For Arabic, it reached a near-perfect 98.9% accuracy.
Key Insights
- Acoustic Similarity: Words that sound similar across different languages naturally cluster together in the model's latent space. Leveraging this similarity significantly boosts accuracy compared to random label mapping.
- Complementary Strengths: While AR alone underperformed standard TL, the combination (AR+TL) yielded the best results. AR handles input-level domain shifts, while TL handles representation-level shifts.
- Stability: Training on very small datasets usually results in high variance. Adding AR to the pipeline notably reduced the standard deviation of the accuracy, making the system more stable and reliable.
Conclusion
This study demonstrates that Adversarial Reprogramming, when combined with Transfer Learning, provides a powerful and stable framework for low-resource speech recognition. Future work will explore combining AR with modern self-supervised learning methods and investigating its application in improving model robustness for diverse audio processing tasks.
Summary updated