Abstract
Modern Hebrew text typically omits Niqqud (diacritics), creating significant phonetic ambiguity for speech synthesis systems. Traditional approaches rely on complex front-ends to predict missing vowels, often leading to error propagation. This paper presents a novel text-to-speech (TTS) system that treats synthesis as a language modeling problem over discrete audio tokens. By training on 4,500 hours of "in-the-wild" podcast data, the model learns to infer correct pronunciation directly from context. Experimental results demonstrate that this diacritic-free approach significantly outperforms existing baselines in both naturalness and speaker similarity.
Introduction
Hebrew presents a unique challenge for text-to-speech (TTS) synthesis. While classical Hebrew uses Niqqud diacritics to denote vowels, modern writing almost entirely omits them. This lack of phonetic information introduces massive ambiguity; for instance, the word מתנה can be read as matana (gift) or mattanah (conditioning) depending on the context.
Standard TTS architectures typically employ a "front-end" to predict these missing diacritics or convert text to phonemes. However, errors in this stage propagate through the pipeline, resulting in unnatural speech. Furthermore, high-quality Hebrew datasets with diacritics are scarce, usually limited to under 30 hours, which hinders the scaling potential of deep learning models.
Methodology
The proposed system adopts a Language Modeling (LM) approach that bypasses the need for diacritics entirely. Instead of explicit phonetic conversion, the model learns the direct mapping between raw text and audio representations.
The architecture consists of several key components:
- Discrete Audio Representation: Raw waveforms are converted into discrete "acoustic tokens" using a neural audio codec (EnCodec) with Residual Vector Quantization (RVQ).
- Two-Stage Modeling: An Auto-Regressive (AR) model predicts the primary layer of acoustic tokens from input text and a 3-second acoustic prompt (enabling zero-shot speaker cloning). Subsequently, a Non-Auto-Regressive (NAR) model generates the remaining acoustic details for high-fidelity output.
- Word-Piece Tokenization: Utilizing a word-piece tokenizer (based on AlephBERT) allows the model to capture richer semantic and morphological context. This is particularly effective for Hebrew, where prefixes and suffixes are common and carry significant grammatical weight.
- Vocoding: A Multi-Band Diffusion (MBD) vocoder translates the predicted tokens back into high-quality raw audio.
Data Scaling and Training
A core contribution of this work is the shift from small, clean datasets to massive "in-the-wild" data. The researchers utilized approximately 4,500 hours of Hebrew speech from the ivrit.ai and HebDB datasets, harvested from local podcasts. Although this data is "weakly supervised" (using automatically generated transcripts), the sheer volume allows the model to learn robust contextual representations that smaller, diacritic-heavy datasets cannot provide. For evaluation, the high-quality 30-hour SASPEECH dataset was used.
Experimental Results
The proposed model was evaluated against Meta’s MMS (Multilingual Speech) and Overflow, a prominent Hebrew-specific baseline.
As shown in the results, the proposed system achieved a Naturalness score of 4.17 in human studies, significantly surpassing Overflow (3.44) and MMS (2.51). In terms of content preservation, the model scored 4.44, indicating high fidelity to the input text. Furthermore, the model demonstrated strong zero-shot voice cloning capabilities, achieving a speaker similarity score of 0.95.
Ablation studies also confirmed that Word-Piece tokenization is superior to character-level tokenization for Hebrew TTS. By identifying word roots and grammatical functions more effectively, word-pieces help the model resolve pronunciation ambiguities that character-level models struggle with.
Discussion and Limitations
While the LM-based approach offers a significant leap in quality, several challenges remain:
- Inference Latency: The auto-regressive nature of the model makes it slower than non-autoregressive "feed-forward" systems.
- Hallucinations: Like other large language models, the system can occasionally skip or invent words, though this was not found to be a major issue in the current evaluation.
- Controllability: The architecture currently offers less direct control over prosodic features like pitch and emotion compared to traditional parametric models.
Future work will focus on constructing even larger, higher-quality Hebrew datasets to further refine the "in-the-wild" training paradigm and improve the model's robustness and speed.
Summary updated