Abstract
Speech-to-singing (S2S) conversion requires precise temporal alignment between spoken lyrics and target singing vocals. This paper presents a hybrid alignment strategy combining model-based forced alignment with feature-based Dynamic Time Warping (DTW). By leveraging reliable speech word boundaries as anchors and refining singing boundaries through a localized search, the proposed method achieves a 47.5% reduction in word boundary error, significantly enhancing the quality of synthesized singing.
Introduction
In speech-to-singing (S2S) conversion, the objective is to transform a spoken recording into a singing version that adheres to a specific melody and rhythm. A fundamental challenge in this process is achieving accurate temporal alignment—mapping specific segments of speech to their corresponding segments in the singing audio.
This task is complicated by significant acoustic and temporal differences between the two modalities. Singing often involves non-linear temporal stretching (e.g., elongated vowels), spectral shifts due to physiological changes like a lowered larynx, and distinct prosodic features such as vibrato and wide pitch ranges. Traditional automated methods, such as pure DTW, often suffer from error accumulation over long sequences, while manual alignment is prohibitively slow.
Proposed Methodology
The authors introduce a multi-stage pipeline that integrates model-based and feature-based techniques to ensure robust alignment.
1. Model-Based Forced Alignment
The process begins with initial word boundary estimation. Spoken lyrics are aligned using the Montreal Forced Aligner (MFA). Simultaneously, singing vocals are aligned using a singing-adapted acoustic model. The core hypothesis is that speech acoustic models provide more reliable "anchor" boundaries than singing models.
2. Singing Word Boundary Correction
To address the lower precision of singing models, a "Window Shifter" component refines the initial singing word boundaries. Using the speech word as a reference, the system evaluates potential singing word boundaries within a ±100ms neighborhood. The optimal shift is determined by minimizing the DTW cost between speech and singing features.
3. Feature-Based Fine Alignment
Following word-level correction, the system performs frame-level alignment. By executing DTW using 39-dimensional Mel-frequency cepstral coefficients (MFCCs) strictly within the established word boundaries, the algorithm prevents the "drift" typically associated with long-sequence feature-based alignment.
Experimental Setup and Datasets
The study utilized a parallel speech-singing corpus from the National University of Singapore's HLT Laboratory. The dataset comprises 80 English songs performed by 8 singers (4 male, 4 female), totaling 2,314 utterances. This represents approximately 116 minutes of speech and 237 minutes of singing. Manual annotations of word boundaries served as the ground truth for evaluation.
Key Results and Discussion
The hybrid approach was evaluated against a standard DTW baseline and a basic forced alignment method.
- Quantitative Accuracy: The Mean Word Boundary Error (WBE) was reduced from 315.50 ms in the baseline to 165.35 ms, representing a 47.5% improvement in alignment precision.
- Subjective Quality: In listening tests, the synthesized singing produced using the proposed alignment achieved a Mean Opinion Score (MOS) of 4.33, compared to 2.89 for the baseline.
- User Preference: Listeners preferred the proposed method in 87.08% of head-to-head comparisons.
The success of this method stems from treating speech boundaries as reliable landmarks and employing a "divide and conquer" strategy that restricts DTW to localized word windows, thereby avoiding global optimization errors.
Conclusion and Future Work
The combination of model-based anchors and feature-based refinement provides a significant leap in automated S2S alignment. While highly effective, the authors note that local word-by-word optimization may not always yield the globally optimal path for an entire sentence. Future research will explore global strategies for boundary correction and improved handling of stylistic musical silences and breaths.
Summary updated