Data Augmentation for Speech-Based Diacritic Restoration

Sara Shatnawi, Sawsan Alqahtani, Shady Shehata, Hanan AldarmakiACL 2024

TL;DR

This paper introduces a novel data augmentation technique for Arabic speech-based diacritic restoration, significantly improving model generalization. They also propose a more efficient, truly multi-modal diacritic restoration model that directly uses speech audio, bypassing the need for an external ASR system.

What Problem Does This Paper Address?

Arabic, a language with over 400 million speakers, presents a unique challenge in its written form: the absence of short vowels and diacritics in most everyday texts. This omission can lead to ambiguity, as a single sequence of consonants can have multiple meanings depending on the diacritics. While diacritics are crucial for accurate pronunciation and understanding, especially in religious texts or for language learning, most Arabic text corpora used for training machine learning models are undiacritized.

Previous approaches to diacritic restoration often relied solely on text, but these models struggle when applied to speech transcripts, leading to significant error rates. Some recent work has incorporated speech audio, but these methods often depend on an external Automatic Speech Recognition (ASR) system, increasing complexity and computational cost. This paper tackles two key issues: the scarcity of labeled data for training diacritic restoration models, especially those incorporating speech, and the inefficiency of existing speech-based methods.

Methods

The paper proposes two main contributions:

  1. Data Augmentation for Speech-Based Diacritic Restoration: To combat data scarcity and improve generalization, the authors developed a data augmentation strategy. This involves:
  2. Random Diacritic Generation: Applying diacritics randomly to each letter in an undiacritized text.
  3. Heuristic Rules: Implementing predefined rules to ensure the resulting diacritized text remains pronounceable, even if not semantically correct. These rules address issues like invalid diacritic placements and combinations.
  4. Text-to-Speech (TTS) Synthesis: Converting the augmented, diacritized text into audio using a TTS model. This creates synthetic speech data that pairs with the augmented transcripts. The goal is to force models to learn diacritic-text relationships from acoustic cues rather than solely relying on textual context, thus reducing overfitting.
  5. Multi-Modal Diacritic Restoration Model: The authors propose a truly multi-modal architecture that directly integrates speech audio with text, eliminating the need for a separate, pre-trained ASR model. This model uses:
  6. Text Encoder: A sequence labeling model (e.g., Transformer or LSTM) to process the undiacritized text.
  7. Speech Feature Extraction: A pre-trained self-supervised model (XLS-R) is used to extract speech features. This model is fine-tuned for ASR on the Arabic speech dataset to better capture relevant acoustic cues.
  8. Cross-Attention Mechanism: This mechanism fuses the text and speech representations, allowing the model to leverage information from both modalities. The query vectors come from the text side, and key/value vectors from the speech side.

Datasets

  • ClArTTS Corpus: Used for training and testing the multi-modal model. It contains approximately 12 hours of recorded speech from a single male speaker with manually diacritized and verified transcripts. A separate 30-minute subset is used for testing.
  • Tashkeela Corpus: Used for pre-training the text-side encoder of the multi-modal model. This corpus contains a large collection of Arabic texts, primarily from classical literature.
  • QASR TTS Dataset: Used for testing. This dataset includes one hour of speech from a male speaker and one hour from a female speaker, originally proposed for TTS purposes.
  • Broadcast News Dataset (Baali et al., 2023): A small manual diacritic dataset used for testing.

Key Insights and Ideas

  • Speech is Key for Diacritic Restoration: Incorporating speech audio significantly improves diacritic restoration performance compared to text-only models, as the acoustic signal provides crucial information for distinguishing diacritics.
  • Data Augmentation Reduces Textual Overfitting: Randomly generating diacritics and synthesizing speech forces models to learn diacritic-text mappings from acoustic patterns, leading to better generalization and reducing reliance on potentially ambiguous textual context.
  • True Multi-Modality Offers Efficiency: A model that directly processes speech and text, without an external ASR pipeline, is more streamlined and computationally efficient, even if it doesn't always match the performance of ASR-augmented systems.
  • XLS-R Fine-tuning Matters: Fine-tuning the XLS-R model for ASR on the target Arabic speech data is crucial for its effectiveness in the multi-modal diacritic restoration task. Freezing these parameters after fine-tuning yielded better results, possibly to prevent catastrophic forgetting on the smaller ClArTTS dataset.
  • ASR Model Quality is a Bottleneck: The performance gap between the multi-modal model and the Text+ASR baseline is largely attributed to the superior ASR performance of the Whisper model used in the latter, highlighting the importance of the underlying speech feature extraction model.

Key Results

  • The proposed data augmentation technique consistently improved diacritic restoration performance across various models and datasets, with absolute reductions in Diacritic Error Rate (DER) ranging from 1% to 5%.
  • The multi-modal model, even without data augmentation, showed improvements over text-only baselines.
  • With data augmentation, the multi-modal model consistently outperformed text-only models and showed more significant improvements than the Text+ASR model in some cases.
  • The multi-modal model achieved comparable performance to text-only baselines while being more integrated and efficient.
  • The Text+ASR model, leveraging the Whisper ASR system, generally outperformed the proposed multi-modal model, but at a higher computational cost.

Limitations and Future Work

  • ASR Performance Gap: A significant limitation is the performance difference between the Whisper ASR model (used in Text+ASR) and the fine-tuned XLS-R model. Future work could focus on optimizing both underlying speech models to achieve similar performance for a fairer comparison.
  • Data Augmentation Scope: The synthetic speech used for data augmentation was generated from the same underlying text. Future work could explore using more varied underlying texts from Modern Standard Arabic to further enhance generalization.
  • Underlying Model Optimization: The paper suggests that optimizing the speech feature extraction model (XLS-R) further, perhaps by pre-training on more general Arabic speech data before fine-tuning on ClArTTS, could improve the multi-modal model's performance to match or exceed the Text+ASR approach.
  • Dataset Limitations: The reliance on classical Arabic datasets for training can lead to poor performance on Modern Standard Arabic. The proposed data augmentation and multi-modal approaches offer promising avenues to mitigate this.

Summary updated

BibTeX
@inproceedings{shatnawi2024data,
  author = {Sara Shatnawi and Sawsan Alqahtani and Shady Shehata and Hanan Aldarmaki},
  title = {{Data Augmentation for Speech-Based Diacritic Restoration}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2024},
}
Figure