TL;DR
This paper presents a novel system for precisely aligning two speech utterances of the same phrase, even when spoken by different people with varying speaking styles. By enhancing the Dynamic Time Warping (DTW) algorithm and employing the WSOLA technique, the system achieves highly accurate temporal synchronization, resulting in natural-sounding modified speech.
What Problem Does This Paper Address?
Accurately synchronizing speech utterances is crucial for various applications, including language education (improving pronunciation) and speech therapy (addressing voice, articulation, or accent issues). The challenge lies in aligning speech that may have significant differences in speaking rate, prosody, and even phonetic nuances between two versions of the same phrase. Existing methods often struggle with these variations, leading to unnatural-sounding results. This paper aims to bridge this gap by developing a robust time synchronization system that can handle these discrepancies and produce high-quality synchronized speech.
Methods
The system comprises two main parts:
- Time Alignment (DTW Enhancement): The core of this part is the Dynamic Time Warping (DTW) algorithm, a well-established technique for finding optimal alignments between sequences. However, to handle the significant differences in speech utterances, the authors introduce two key modifications to the standard DTW:
- Accumulated Local Penalty Constraint: This constraint is designed to control the curvature of the time warping path. It penalizes long horizontal or vertical subpaths, which can lead to unnatural repetitions or omissions of speech segments. The penalty is made dependent on whether a segment is classified as speech or silence, allowing for natural pauses to be extended while discouraging unnatural repetitions of speech.
- Smoothing of the Time Warping Vector: To prevent issues with WSOLA when dealing with very slow speech segments (long horizontal subpaths in DTW), a smoothing stage is applied. This ensures that the time warping vector has a minimum slope for speech segments, preventing excessive repetition. For silent segments, horizontal subpaths are retained to allow for natural pause extension.
- Time Scale Modification (WSOLA): The Waveform Similarity Overlap and Add (WSOLA) algorithm is used to modify the time scale of one utterance (the source signal) to match the other (the reference signal), guided by the time warping vector generated by DTW. WSOLA is chosen for its ability to perform time-scale modification without requiring pitch estimation, which simplifies the process and contributes to high-quality output. To further improve the naturalness of the synthesized speech, especially for unvoiced segments, the authors implement a technique where every third consecutive unvoiced segment is reversed in time.
Datasets
The paper mentions using utterances from the TIMIT database as reference signals. For evaluation, they recorded new utterances from male and female speakers and then processed these using their system. They also mention using these recorded utterances in conjunction with TIMIT utterances for listening tests.
Key Insights and Ideas
- DTW's limitations are addressable: While DTW is powerful, its standard form struggles with significant variations in speech. The paper demonstrates that carefully designed constraints and post-processing steps can make it robust enough for challenging time synchronization tasks.
- Speech vs. Silence matters for alignment: Differentiating between speech and silence is crucial for controlling the time warping path. This allows for natural pause extension while preventing unnatural segment repetition during speech.
- WSOLA benefits from DTW enhancements: The WSOLA algorithm, while effective, can produce artifacts with certain time warping paths. The DTW modifications (smoothing and penalty constraints) directly address these limitations, leading to better WSOLA performance.
- Unvoiced segment reversal improves naturalness: A simple yet effective trick of reversing unvoiced segments improves the perceived quality of the synthesized speech, addressing a known issue with WSOLA.
Key Results
- Listening Test Preference: In an A-B preference test, the authors' time synchronization system was clearly preferred over natural synchronization in most cases. This indicates that the system's alignment is perceived as more accurate.
- Qualitative Quality: While the qualitative ratings for speech quality were somewhat inconsistent across listeners, the average rating indicated that the time-modified sentences were experienced as being of good quality, often rated better than the original.
- Robustness: The system is capable of aligning utterances that differ severely, even when spoken by different speakers with different speaking styles, a significant improvement over earlier systems.
Limitations and Future Work
- Inconsistent Quality Judgments: The authors acknowledge that the qualitative judgment of speech quality can be inconsistent among listeners, possibly due to the influence of prosody changes.
- Further Prosody Control: While the system achieves good synchronization, further work could explore more sophisticated control over prosody to enhance naturalness even further.
- Automatic Prosody Adjustment: The paper hints at the prosody being automatically changed by time scaling. Future work could focus on explicitly controlling and adapting prosody to match the reference signal more closely.
Summary updated