基於時域上基週同步疊加法之歌聲合成系統 (Singing Voice Synthesis System Based on Time Domain-Pitch Synchronized Overlap and Add) [In Chinese]

Ming-Kuan Wu, Chia-Ping ChenROCLING 2013

Abstract

This paper presents a concatenation-based singing voice synthesis (SVS) system designed for Mandarin Chinese. The system utilizes the Time Domain-Pitch Synchronized Overlap and Add (TD-PSOLA) algorithm to achieve precise control over pitch and duration. By integrating a custom multi-pitch syllable corpus with melody data extracted from MIDI files, the system produces synthesized singing that blends naturally with background music. Evaluation results indicate that the inclusion of accompaniment significantly enhances perceived vocal quality.

Introduction

Singing voice synthesis (SVS) presents unique challenges compared to standard text-to-speech (TTS). While TTS focuses on intelligibility and natural prosody, SVS requires strict adherence to musical parameters: hitting exact pitches, sustaining notes for specific durations, and managing volume dynamics—all while preserving the singer's unique timbre. This research addresses the "robotic" nature of simple concatenation by focusing on smooth pitch transitions and the elimination of boundary artifacts (clicks) between syllables.

System Architecture and Methodology

The system follows a concatenation-based architecture where pre-recorded speech units are joined to form a song. The core components include:

  • TD-PSOLA Engine: Operating directly in the time domain, TD-PSOLA uses Hanning windows to isolate pitch periods. These periods are shifted and overlapped to modify the fundamental frequency ($F_0$) or duration without distorting the vocal tract characteristics (timbre).
  • Information Extraction: The system parses MIDI files to extract musical scores, including note numbers, velocities (volume), and timing. Lyrics are mapped to Mandarin syllables using a Pinyin/Zhuyin conversion system.
  • Pitch Transition Modeling: To mimic human vocal constraints, a Sigmoid function is employed to create natural pitch glides (portamento) between notes, avoiding jarring frequency jumps.
  • Boundary Processing: To reduce concatenation artifacts, a linear fade-out is applied to the end of each synthesized syllable before joining it to the next.
Detailed system flowchart for Mandarin singing voice synthesis

Corpus Construction and Normalization

A specialized Mandarin syllable corpus was recorded by a male singer to support the synthesis engine. The corpus includes all standard Mandarin syllables recorded at three distinct pitches (MIDI note numbers 44, 50, and 56). This multi-pitch approach ensures that the TD-PSOLA algorithm does not need to shift samples excessively, thereby preserving a natural tone.

Because the corpus was recorded over multiple sessions, volume normalization was critical. The researchers implemented a robust normalization process to ensure a consistent baseline for the synthesis engine. The distribution of the 1,242 segments (414 syllables × 3 pitches) and their maximum volume values are analyzed to ensure uniformity.

Maximum volume distribution of the 1,242 syllable segments in the corpus

Experimental Results

The system was evaluated by 10 participants using a Mean Opinion Score (MOS) across various genres.

  • Quality and Clarity: The average quality score for "dry" (unaccompanied) synthesis was 2.4, which improved to 2.8 when background music was added. Clarity and similarity scores averaged 2.6 and 2.5, respectively.
  • Genre Performance: The system excelled in slow-tempo genres like nursery rhymes and folk songs. However, it struggled with fast-paced R&B tracks, where rapid syllable changes made the concatenation boundaries more perceptible.

Conclusion and Future Work

The proposed system demonstrates the effectiveness of TD-PSOLA for Mandarin SVS, particularly when paired with background music which provides a "masking" effect for minor artifacts. Future improvements will focus on phonetic granularity—splitting syllables into consonants and vowels for better duration control—and modeling expressive features like vibrato and breathiness to enhance emotional fidelity.

Summary updated

BibTeX
@inproceedings{wu2013singing,
  author = {Ming-Kuan Wu and Chia-Ping Chen},
  title = {{基於時域上基週同步疊加法之歌聲合成系統 (Singing Voice Synthesis System Based on Time Domain-Pitch Synchronized Overlap and Add) [In Chinese]}},
  booktitle = {ROCLING},
  year = {2013},
}
Figure