Enhancing TTS Stability in Hebrew using Discrete Semantic Units

Ella Zeldes, Or Tal, Yossi Adi

TL;DR

Researchers from The Hebrew University of Jerusalem have developed LOTHM (Language of The Hebrew Man), a Text-to-Speech (TTS) system that significantly improves stability for non-diacriticized languages like Hebrew. By switching from traditional acoustic tokens to discrete semantic units (HuBERT codes), the model achieves a 2x improvement in Word Error Rate (WER) and eliminates the need for multiple "best-of-k" sampling attempts to get a clean result.

What Problem Does This Paper Address?

Most modern TTS systems struggle with Hebrew because of a linguistic quirk: vowel pointing. In standard Hebrew, vowels are rarely written; readers must infer them from context. For a computer, the word "L-M-D" could mean "learned," "teaches," or "was taught" depending on the missing diacritics.

Previous solutions either relied on "Nakdan" tools (pre-processors that guess the vowels) or used acoustic tokenizers like EnCodec. However, acoustic tokens are optimized for audio reconstruction, not phonetic meaning. This often leads to "unstable" generations—speech that sounds natural but hallucinates words or gets the pronunciation wrong. To fix this, researchers usually have to generate several samples (best-of-k sampling) and pick the best one, which is computationally expensive and slow.

Methods

The authors introduce LOTHM, which moves away from acoustic reconstruction in favor of semantic representation.

  1. Semantic Tokenization: Instead of EnCodec, the model uses HuBERT (Hidden-Unit BERT) units. These are extracted from the 12th layer of a self-supervised model and clustered into 2,000 discrete centroids. Because HuBERT is trained to predict masked speech, its tokens correlate much more strongly with phonetic content than acoustic details.
  2. The Architecture:
    • Autoregressive Language Model (AR-LM): A Transformer-based model (similar to VALL-E) that predicts the sequence of HuBERT tokens based on a text prompt and a short (3-second) audio prompt.
  3. Unit-based Vocoder: Since HuBERT tokens discard speaker identity to focus on meaning, the system uses a modified HiFi-GAN vocoder. This vocoder is conditioned on a separate speaker embedding (ECAPA-TDNN) to re-inject the specific vocal characteristics of the target speaker.
  4. Zero-Shot Capability: By separating the "what is said" (HuBERT) from "who is saying it" (Speaker Embedding), the model can synthesize speech for speakers it has never seen during training.

Datasets

The model was trained on a massive multi-lingual corpus:

  • Hebrew: ~4,500 hours of audio from HebDB and ivrit.ai, primarily sourced from podcasts featuring spontaneous, multi-speaker dialogues.
  • English: ~44 hours from VCTK and 585 hours from LibriTTS to create a robust, multi-lingual vocoder.
  • Evaluation: The team used the SASPEECH dataset (a high-quality single-speaker Hebrew dataset) to benchmark against existing models like MMS and HebTTS.

Key Insights and Ideas

  • Semantic > Acoustic for Stability: The "aha moment" is that semantic units act as a natural bridge for missing vowels. Because HuBERT tokens capture the underlying phonetic structure, the model is less likely to "guess" a wrong pronunciation compared to models trying to predict raw acoustic waveforms.
  • The "Factor of 2" Improvement: Using HuBERT codes improved the Word Error Rate (WER) and Character Error Rate (CER) by roughly 50% compared to EnCodec-based models.
  • One-Shot Success: While previous models like HebTTS required "best-of-3" sampling to reach acceptable quality, LOTHM’s single-sample performance (x1) actually outperformed the baseline's best-of-3 performance.
  • Disentanglement: The architecture successfully separates speaking rate and speaker identity. You can provide a fast-talking prompt and a slow-talking speaker embedding, and the model can control these variables independently.

Key Results

  • Stability: On Hebrew tasks, LOTHM achieved a WER of 0.17 on a single sample, compared to 0.34 for the EnCodec-based HebTTS.
  • English Adaptability: The benefits aren't limited to Hebrew. When tested on English (LibriTTS), HuBERT tokens consistently outperformed EnCodec in transcription accuracy (WER 0.15 vs 0.22).
  • Human Evaluation: In "Mean Opinion Score" tests, native Hebrew speakers rated LOTHM's content match and naturalness on par with or better than specialized diacritic-based models, despite LOTHM not using any explicit vowel markers.

Limitations and Future Work

  • Prosody and Emotion: While the model is stable and natural, the authors note that future work could focus on better integration of emotion and intonation to make the speech even more expressive.
  • Phonetic Complexity: The current approach works well for Hebrew and English, but the authors suggest extending it to other languages with even more complex phonetic structures or different types of non-diacriticized scripts.
  • Vocoder Quality: While the unit-based vocoder is efficient, there is always room to improve the "crispness" of the audio to match high-fidelity studio recordings.

Summary updated

BibTeX
@inproceedings{zeldes2025enhancing,
  author = {Ella Zeldes and Or Tal and Yossi Adi},
  title = {{Enhancing TTS Stability in Hebrew using Discrete Semantic Units}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888310},
}
Figure