SeDepTTS: Enhancing the Naturalness via Semantic Dependency and Local Convolution for Text-to-Speech Synthesis

Chenglong Jiang, Ying Gao, Wing W. Y. Ng, Jiyong Zhou, Jinghui Zhong, Hongzhong ZhenAAAI 2023 · DOI

Abstract

Non-autoregressive text-to-speech (TTS) models like FastSpeech have achieved high synthesis speeds but often struggle with natural prosody. SeDepTTS addresses this by incorporating semantic dependency information and local convolutional structures. By utilizing a Semantic Dependency Extractor (SDE) to generate Gaussian priors and replacing linear layers in attention heads with 1-D convolutions, the model better captures both global context and local dependencies. Experimental results on the BIAOBEI dataset demonstrate that SeDepTTS significantly improves Mean Opinion Scores (MOS) and pitch accuracy compared to baseline models.

Introduction

The transition from autoregressive to non-autoregressive TTS models has prioritized speed, often at the expense of the subtle "human" elements of speech—pauses, stress, and intonation. The primary limitation of standard Transformer-based models is the self-attention mechanism, which can disperse focus too broadly. In speech synthesis, local context—the relationship between a word and its immediate neighbors or its grammatical role—is crucial for natural prosody. SeDepTTS introduces a hybrid approach to bridge this gap by teaching the model how words relate semantically.

Methodology

Model Architecture

SeDepTTS builds upon the FastSpeech2 framework, introducing two key modifications: the Semantic Dependency Extractor (SDE) and Local Convolution (LC) within the Feed-Forward Transformer (FFT) blocks.

Architecture of SeDepTTS and the modified FFT Block featuring local convolutions.

Semantic Dependency Extractor (SDE)

The SDE block moves beyond simple phoneme sequences by analyzing the Semantic Dependency Graph (SDG) of the input text. It identifies meaningful links between words (e.g., an adjective and the noun it describes) and converts these relationships into a semantic dependency matrix. This matrix serves as a Gaussian prior, providing a mathematical "nudge" to the attention mechanism. This allows the model to maintain multiple centers of focus in complex sentences, ensuring that "breath groups" and semantic clusters are handled naturally.

The Semantic Dependency Extractor (SDE) block and the resulting semantic dependency matrix used to guide attention.

Local Convolution (LC)

To further enhance local context, SeDepTTS replaces the standard linear layers used for generating Query (Q) and Value (V) vectors in the multi-head attention mechanism with 1-D Convolutional Neural Networks (CNNs). Convolutions inherently focus on neighboring data points, forcing the model to integrate local contextual information directly into the attention process, ensuring that nearby characters influence each other's sound.

Dataset

The model was trained and evaluated using the BIAOBEI dataset, a widely recognized open-source Mandarin Chinese dataset. It contains 10,000 sentences (roughly 12 hours of audio) recorded by a single professional female speaker.

Results and Discussion

SeDepTTS demonstrates superior performance across both subjective and objective metrics:

  • Subjective Quality: In Mean Opinion Score (MOS) evaluations, SeDepTTS achieved a score of 4.10, a significant improvement over FastSpeech2's 3.72 and even surpassing some older autoregressive models.
  • Prosody Modeling: The model's pitch contour $R^2$ (goodness of fit) reached 0.838, compared to 0.784 for FastSpeech2, indicating a much closer match to the human original.
  • Visual Analysis: Mel spectrogram analysis confirmed that SeDepTTS was the only model to correctly replicate the specific pauses and falling/rising intonations found in the ground-truth human recordings.

The integration of semantic dependency graphs proves that meaning-based relationships are a better guide for prosody than simple syntactic rules. By "sharpening" the attention focus with Gaussian priors, the model effectively corrects the "blur" often found in standard self-attention distributions.

Conclusion

SeDepTTS successfully enhances the naturalness of non-autoregressive TTS by combining global semantic understanding with local acoustic precision. While currently dependent on external linguistic tools like jieba and ltp for Chinese processing, the system sets a new standard for prosodic accuracy. Future work will focus on mining deeper semantic information to handle highly expressive tasks, such as professional poetry recitation, where rhythmic timing and emotional weight are critical.

Summary updated

BibTeX
@inproceedings{jiang2023sedeptts,
  author = {Chenglong Jiang and Ying Gao and Wing W. Y. Ng and Jiyong Zhou and Jinghui Zhong and Hongzhong Zhen},
  title = {{SeDepTTS: Enhancing the Naturalness via Semantic Dependency and Local Convolution for Text-to-Speech Synthesis}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26523},
}
Figure