Abstract
High-fidelity text-to-speech (TTS) synthesis for under-resourced languages remains a significant challenge. This paper introduces Bahasa Harmony, a professionally recorded 55-hour dataset for the Bahasa language, and EnGen-TTS, a novel architecture designed for efficient and natural speech generation. By employing discrete codec modeling and a multi-lingual T5 encoder, EnGen-TTS achieves a state-of-the-art Mean Opinion Score (MOS) of 4.45 and exceptional inference speeds with a Real-Time Factor (RTF) of 0.016. Our evaluations demonstrate that the model not only excels in Bahasa but also exhibits remarkable zero-shot capabilities in cross-lingual tasks.
Introduction
Bahasa, encompassing Indonesian and Malay, is spoken by hundreds of millions yet lacks high-quality, open-source TTS resources. Existing systems often fail to capture the phonetic nuances and cultural registers of the language. Moreover, many high-performance TTS models are computationally intensive, hindering real-time deployment. To address these issues, we present a robust dataset and an efficient model architecture that balances linguistic richness with high-speed performance.
The EnGen-TTS Architecture
EnGen-TTS is built upon a discrete codec modeling framework, inspired by the "Bark" model, which treats speech synthesis as a sequence-to-sequence problem. The architecture integrates several key components:
- Text Encoding: A frozen m-T5 (multi-lingual T5) encoder extracts semantic latents ($X_{te}$) from input text ($X_t$).
- Audio Discretization: The system utilizes a frozen EnCodec to represent audio as discrete tokens, simplifying the generation process into predicting token sequences.
- Neural Codec Language Model ($N_c$): An autoregressive module that predicts audio tokens based on text latents and speaker embeddings ($X_{se}$) from a dedicated Speaker Encoder.
- Vocoding: A fine-tuned HiFi-GAN vocoder ($V_c$) converts the predicted tokens back into high-fidelity waveforms.
The model is optimized using a composite loss function comprising cross-entropy ($L_{ce}$), Mel-spectrogram loss ($L_{mel}$), and GAN-based adversarial loss ($L_{gan}$). By leveraging frozen pre-trained components, the authors significantly reduce the computational burden of training without sacrificing the ability to capture complex linguistic features.
The Bahasa Harmony Dataset
The Bahasa Harmony Dataset was developed to provide a high-quality foundation for Bahasa TTS. It consists of approximately 55 hours of audio across 52,000 recordings, featuring:
- Diverse Vocabulary: Over 23,000 unique terms sourced from Wikipedia and conversational translations.
- Professional Quality: Recorded at 48 kHz in controlled environments by two professional voice artists (male and female) from Southern Indonesia.
Experimental Results
EnGen-TTS demonstrates superior performance in both quality and efficiency. It achieved a MOS of 4.45, outperforming established baselines like NaturalSpeech (4.39) and CLaM-TTS (4.41). In terms of speed, the model reaches a Real-Time Factor (RTF) of 0.016, enabling the synthesis of one second of audio in just 16 milliseconds.
Surprisingly, EnGen-TTS also showed strong zero-shot performance. In cross-lingual evaluations involving seven languages, the model achieved higher MOS in Hindi (4.55) and Marathi (4.87) than models specifically trained for those languages, highlighting the robustness of the m-T5 encoder and the codec-based approach.
Discussion and Future Work
While EnGen-TTS sets a new benchmark for Bahasa TTS, certain limitations remain. The current model is optimized for 22.05 kHz; downsampling to 8 kHz for telephony applications results in a loss of clarity. Additionally, the context window is currently limited to 500 audio tokens (~16 seconds), which can lead to "catastrophic forgetting" in longer narrations. Future work will focus on direct 8 kHz generation and extending the context window to 2048 tokens to support paragraph-level synthesis.
Summary updated