TL;DR
This paper investigates how the proportion of different Catalan dialects in training data affects automatic speech recognition (ASR) system robustness to both dialectal variation and adversarial attacks. They found that balancing dialects improves ASR performance across dialects but does not significantly enhance robustness against adversarial noise, suggesting a potential security vulnerability in current ASR systems.
What Problem Does This Paper Address?
Automatic Speech Recognition (ASR) systems are increasingly expected to perform well across a wide range of speech variations, including accents and dialects. While incorporating diverse dialects into training data is a common strategy to improve robustness, the optimal way to balance these dialects remains an open question. Furthermore, the susceptibility of ASR models to adversarial attacks, which are designed to intentionally disrupt performance, is a growing concern. This paper specifically addresses whether manipulating the dialect composition of training data for Catalan ASR can simultaneously improve robustness to dialectal variation and adversarial perturbations, or if these two aspects of robustness are independent or even conflicting.
Methods
The researchers fine-tuned a pre-trained XLSR-53 model for Catalan ASR. They experimented with four different dialect compositions for the fine-tuning data, varying the proportion of the Central Catalan dialect from 100% down to 20%, with the remaining data split evenly among four other Catalan dialects (Balearic, Nord, Nord-Occidental, and Valencià).
To evaluate ASR performance, they measured Word Error Rate (WER) and Character Error Rate (CER) on test sets representing the five dialects. For adversarial robustness, they employed Carlini & Wagner (CW) attacks, a common method for generating adversarial perturbations. These attacks aim to alter the input audio such that the ASR model produces an incorrect transcription while keeping the perturbation imperceptible (measured by Signal-to-Noise Ratio, SNR).
Datasets
The study utilized existing Catalan speech data from the Common Voice corpus. The fine-tuning data was curated to represent five distinct Catalan macro-dialects: Balearic, Central, Nord, Nord-Occidental, and Valencià. The specific proportions of these dialects were manipulated for the four experimental conditions.
Key Insights and Ideas
- Dialect Balancing for ASR is Beneficial, but Not Necessarily Perfect: While increasing the representation of non-Central dialects improved overall ASR performance across various Catalan dialects, achieving perfect balance wasn't strictly necessary for substantial gains.
- Dialect Diversity Doesn't Guarantee Adversarial Robustness: A key finding is that improving dialectal robustness through data diversification did not translate into improved resistance against adversarial attacks. This suggests that ASR models might remain vulnerable to malicious manipulation even when trained on a diverse set of dialects.
- Potential for Security Vulnerabilities: The lack of improved adversarial robustness implies that current ASR systems, even those fine-tuned with dialectal diversity, could still be susceptible to attacks that alter their behavior, posing a security risk.
- Dialect-Specific Performance Nuances: The study highlighted that some dialects (e.g., Balearic) might require more targeted attention or up-sampling in future corpus compilation efforts to achieve comparable performance to other dialects.
Key Results
- ASR Performance: As the proportion of Central Catalan in the fine-tuning data decreased (i.e., more balanced dialect representation), the Word Error Rate (WER) and Character Error Rate (CER) generally improved across all tested dialects. The 20% and 50% Central Catalan models showed the most consistent improvements.
- Adversarial Attacks: The study found a high success rate for CW attacks across all tested model configurations, indicating that even high-performing models are susceptible. Crucially, the proportion of Central Catalan in the fine-tuning data had no significant impact on the success rate or average SNR of these adversarial attacks. The average SNR for successful attacks was consistently high across all conditions, suggesting the perturbations were largely imperceptible.
Limitations and Future Work
The authors acknowledge several limitations:
- Language Specificity: The study focused solely on Catalan dialects. The findings might differ for languages with dialects that are less mutually intelligible, where cross-dialect transfer might be more challenging.
- Data Collection Challenges: Creating large, multilingual, transcribed speech datasets with accurate accent annotations is a significant hurdle.
- Computational Constraints: The fine-tuning was performed only once per data composition. Repeating experiments with multiple random seeds and exploring other ASR architectures (including larger versions of XLSR) and languages would be beneficial.
Future work suggested includes:
- Investigating how orthographic transparency of languages impacts adversarial vulnerability in ASR.
- Further research into multilingual and multi-dialect speech processing models to identify specific vulnerabilities arising from unbalanced datasets or mismatched labels and audio.
- Exploring other ASR architectures and a wider range of languages and their dialects.
Summary updated