Enhancing TTS Stability in Hebrew using Discrete Semantic Units

Ella Zeldes, Or Tal, Yossi Adi

Abstract

Hebrew Text-to-Speech (TTS) systems face significant stability challenges due to the language's non-diacriticized nature, where vowels are typically omitted in writing. This paper introduces LOTHM (Language of The Hebrew Man), a TTS framework that replaces traditional acoustic tokens with discrete semantic units (HuBERT codes). By leveraging semantic representations, LOTHM achieves a twofold improvement in Word Error Rate (WER) and provides high-quality "one-shot" synthesis, removing the computational overhead of "best-of-k" sampling.

Introduction

Modern TTS systems often struggle with Hebrew because of its reliance on context for vowel inference—a linguistic feature known as vowel pointing. Without explicit diacritics, a single consonant sequence can represent multiple words with distinct pronunciations. Previous approaches relied on error-prone pre-processors or acoustic tokenizers like EnCodec. However, acoustic tokens prioritize audio reconstruction over phonetic accuracy, leading to "unstable" generations characterized by word hallucinations or mispronunciations.

Methodology

LOTHM shifts the focus from acoustic reconstruction to semantic representation. The architecture is composed of three primary modules:

  1. Semantic Tokenization: The system utilizes HuBERT (Hidden-Unit BERT) units extracted from the 12th layer of a self-supervised model. These units are clustered into 2,000 discrete centroids. Unlike acoustic tokens, HuBERT units capture phonetic structure, making them more resilient to the ambiguities of non-diacriticized text.
  2. Autoregressive Language Model (AR-LM): A Transformer-based model predicts the sequence of HuBERT tokens based on a text prompt and a 3-second audio prompt.
  3. Unit-based Vocoder: To restore speaker identity lost during semantic tokenization, a modified HiFi-GAN vocoder is used. It is conditioned on a separate speaker embedding (ECAPA-TDNN) to re-inject the target speaker's vocal characteristics.
Figure 1: The LOTHM architecture, showing the integration of text prompts, HuBERT semantic units, and speaker embeddings into the auto-regressive LM and vocoder.

Experimental Setup

The model was trained on approximately 4,500 hours of Hebrew audio (HebDB and ivrit.ai) and over 600 hours of English audio (VCTK and LibriTTS). Evaluation was conducted using the SASPEECH dataset, comparing LOTHM against established models like MMS and HebTTS.

Results and Discussion

The transition to semantic units yielded substantial improvements in both objective and subjective metrics.

  • Stability and Accuracy: LOTHM achieved a WER of 0.17 on a single sample (x1), a 50% reduction compared to the 0.34 WER of the EnCodec-based HebTTS. Notably, LOTHM's single-sample performance outperformed the "best-of-3" performance of previous baselines.
  • Human Evaluation: Native speakers rated LOTHM higher in naturalness and content accuracy. The model successfully synthesized speech that matched the target speaker's identity even in zero-shot scenarios.
Figure 3: Objective and human study results comparing LOTHM to baseline models. LOTHM (x1) shows a significant reduction in WER and CER compared to HebTTS (x1).

Conclusion

LOTHM demonstrates that discrete semantic units are superior to acoustic tokens for stabilizing TTS in languages with complex phonetic-orthographic relationships like Hebrew. By disentangling content (HuBERT) from identity (Speaker Embedding), the system provides a robust, efficient, and high-fidelity solution for zero-shot speech synthesis. Future work will explore enhancing emotional expressiveness and extending the approach to other phonetically complex languages.

Summary updated

BibTeX
@inproceedings{zeldes2025enhancing,
  author = {Ella Zeldes and Or Tal and Yossi Adi},
  title = {{Enhancing TTS Stability in Hebrew using Discrete Semantic Units}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2025},
  doi = {10.1109/icassp49660.2025.10888310},
}
Figure