TL;DR
This paper presents the first Text-to-Speech (TTS) system and dataset for Shipibo-Konibo, a low-resource indigenous language. The developed system achieved impressive intelligibility and naturalness scores, even surpassing natural speech in some evaluations, highlighting the potential for TTS in language documentation and revitalization.
What Problem Does This Paper Address?
The world is home to over 7,000 languages, but a significant portion are endangered, threatening linguistic diversity and cultural heritage. For many of these low-resource languages, particularly indigenous ones, the lack of digital resources like speech datasets severely hinders the development of Natural Language Processing (NLP) tools. This gap makes it difficult to create applications that could aid in language documentation, preservation, and revitalization. Specifically, the Shipibo-Konibo language, spoken by around 40,000 people in the Peruvian Amazon, faces challenges due to multiple orthographic traditions and a scarcity of digital resources, making it a prime candidate for developing low-resource NLP solutions.
Methods
The researchers tackled this challenge by building a TTS system using a two-stage approach:
- Tacotron 2 for Spectrogram Prediction: They selected Tacotron 2, a well-established encoder-decoder architecture, known for its effectiveness in low-resource TTS. To overcome data scarcity, they employed transfer learning, fine-tuning a pre-trained Latin American Spanish model. This allowed them to leverage existing knowledge and adapt it to the Shipibo-Konibo language.
- HiFi-GAN for Waveform Generation: For converting the predicted spectrograms into audible speech, they utilized HiFi-GAN, a Generative Adversarial Network (GAN)-based vocoder. HiFi-GAN was chosen for its superior performance in generating high-fidelity audio waveforms, offering an improvement over older vocoders like WaveGlow.
The training process involved fine-tuning a pre-trained Spanish model on the newly created Shipibo-Konibo dataset.
Datasets
- Shipibo-Konibo Dataset (New Benchmark): The core contribution is a newly created dataset for Shipibo-Konibo.
- Text Compilation: Over 3,025 sentences were compiled from sources like "The Little Prince" translated into Shipibo-Konibo and educational materials from the Peruvian Ministry of Education.
- Audio Compilation: Approximately 4 hours of audio recordings were collected from a single native Shipibo-Konibo speaker, adhering to standards similar to the LJ Speech dataset (22.05kHz sampling rate, single speaker, diverse phonemes, 1-10 second duration, minimal silence).
- Evaluation Data: For evaluation, an additional 400 sentences were extracted from a pre-2015 Shipibo-Konibo book, "Koshi Shinanya Ainbo," to ensure relevance and adaptability to the modern alphabet.
Key Insights and Ideas
- Transfer Learning is Crucial: Fine-tuning a pre-trained model from a related language (Spanish) significantly boosts performance in low-resource scenarios, demonstrating the power of leveraging existing linguistic knowledge.
- Synthetic Speech Can Outperform Natural Speech: Surprisingly, the synthetic Shipibo-Konibo voice achieved higher intelligibility and naturalness scores than the natural recordings in evaluations. This suggests that with careful data curation and model training, synthetic voices can be highly effective, even for languages with limited natural recordings.
- TTS for Language Revitalization: The successful development of this TTS system underscores its potential as a tool for language documentation and revitalization, enabling pronunciation practice and enhancing accessibility in educational materials.
- Qualitative Analysis Reveals Nuances: While quantitative results were strong, qualitative analysis highlighted areas for improvement, such as intonation and punctuation handling, pointing to the need for more diverse training data and potentially more sophisticated model architectures for certain linguistic features.
Key Results
- Intelligibility Rate (IR): The synthetic voice achieved an IR of 88.56%, surpassing the natural voice's IR of 83.45%.
- Mean Opinion Score (MOS): The synthetic voice received an average MOS of 4.01 ± 1.09, compared to the natural voice's 3.75 ± 1.2. This indicates a higher perceived quality and naturalness for the synthesized speech.
- Dataset Size: The final dataset comprised over 4 hours of audio and 3,025 written sentences.
- Training: The Tacotron 2 model was trained for approximately 225 epochs, and the HiFi-GAN vocoder was trained for 34 epochs.
Limitations and Future Work
Limitations:
- Corpus Size: The 4-hour corpus might not fully capture the phonetic and prosodic variability of Shipibo-Konibo, potentially limiting generalization.
- Voice Diversity: Training on a single speaker limits voice diversity; multi-speaker training is recommended for broader applicability.
- Speaker's Pronunciation: The native speaker, while valuable, had been living away from her community for several years, which may have introduced subtle pronunciation variations.
- Lack of Objective Metrics: Due to the absence of high-quality reference data, standard objective metrics like PESQ and POLQA could not be used, relying instead on subjective evaluations.
- Orthographic Variations: Shipibo-Konibo has multiple writing conventions, and the dataset prioritized one standard, which might affect usability in communities using different conventions.
Future Work:
- Improve Corpus Quality: Focus on refining audio recording conditions and expanding the dataset.
- Incorporate More Diverse Sentence Structures: To address issues like ambiguous intonation and punctuation handling.
- Adapt to Other Languages: The developed framework can serve as a foundation for adapting TTS systems to other Panoan or Amazonian languages.
- Multi-Speaker Training: To increase voice diversity and model robustness.
Summary updated