TL;DR
This paper introduces a novel end-to-end statistical parametric speech synthesis system that leverages multiple feed-forward Deep Neural Networks (DNNs) to handle all stages of the synthesis pipeline, from acoustic feature extraction to spectral post-filtering. By integrating these DNNs, the system aims to achieve higher quality speech synthesis compared to conventional methods and single-DNN approaches.
What Problem Does This Paper Address?
Statistical Parametric Speech Synthesis (SPSS) has seen significant improvements with the advent of Deep Neural Networks (DNNs). While DNNs have individually enhanced components like acoustic modeling, spectral feature extraction, and post-filtering, this paper addresses the challenge of effectively integrating these DNN-powered components into a cohesive, high-quality end-to-end system. The goal is to move beyond optimizing individual modules and instead create a system where DNNs handle the entire synthesis process, aiming for better synergy and overall performance.
Methods
The core of the proposed system is the combination of three types of feed-forward DNNs:
- Acoustic Feature Extraction: A deep auto-encoder is used for data-driven, non-linear feature extraction from spectral amplitudes. This auto-encoder is pre-trained layer-by-layer and then fine-tuned to minimize reconstruction error.
- Spectral Amplitude Modeling: A DNN-based acoustic model is trained to directly predict spectral amplitudes from linguistic features. This involves a pre-training technique where a deep auto-encoder extracts bottleneck features, which are then used to train a separate DNN acoustic model. Finally, the auto-encoder's decoder and the acoustic model are stacked and fine-tuned together.
- Spectral Post-filter: A feed-forward DNN is employed as a post-filter to enhance spectral amplitudes. This DNN takes consecutive frames as input and output, allowing it to smooth spectral differences and enhance spectral peaks in the time-frequency domain, aiming to reconstruct fine spectral structures lost during modeling.
The system integrates these components sequentially, with the output of one DNN serving as the input for the next, creating an end-to-end pipeline.
Datasets
- English Database: The Blizzard Challenge 2011 dataset, containing approximately 17 hours of speech data and 12K utterances. Linguistic information was automatically generated using Festival.
- Korean Database: A proprietary database of approximately 38 hours of speech data and about 12K utterances, primarily consisting of news sentences. Phoneme sequences and boundaries were manually corrected.
The paper notes that the test sentences were not included in the training utterances for both datasets.
Key Insights and Ideas
- End-to-End DNN Integration: The paper demonstrates the feasibility and potential benefits of using multiple DNNs to cover the entire speech synthesis pipeline, moving towards a more unified neural approach.
- Auto-encoder for Feature Extraction: Utilizing a deep auto-encoder for spectral feature extraction offers a data-driven way to learn compressed representations, potentially improving downstream modeling.
- Post-filtering for Spectral Enhancement: The DNN-based post-filter is crucial for refining the synthesized spectral amplitudes, addressing spectral smoothing and peak enhancement to improve naturalness.
- Language-Specific Performance Differences: The study highlights that the effectiveness of the proposed DNN combination can vary between languages (English vs. Korean), suggesting that architectural choices and data characteristics (like sampling rate and spectral analysis conditions) play a significant role.
Key Results
- English Experiment: The proposed DNN combination system (MDNNs1 and MDNNs2) was evaluated as better than conventional HMM, single DNN, and even unit selection systems in subjective evaluations.
- Korean Experiment: The results were mixed. The proposed system performed comparably to HMM and slightly worse than the single DNN system, which was an "opposite outcome" to the English findings.
- Subjective Evaluation: MUSHRA tests were conducted with native speakers. The paper reports statistical significance for the improvements observed in the English experiment.
Limitations and Future Work
- Language Dependency: The authors acknowledge that the performance differences between English and Korean experiments might stem from architectural variations due to different FFT points and sampling rates. Further investigation into 16kHz sample rate speech is suggested.
- F0 and Aperiodicity: The current proposed system synthesizes only spectral features, relying on HMM or DNN for F0 and aperiodicity. Future work includes investigating the modeling of F0 and aperiodicity measures within the proposed DNN framework.
- Architectural Exploration: The authors plan to explore the effect of different DNN structures more thoroughly.
Summary updated