A Combination of Model-Based and Feature-Based Strategy for Speech-to-Singing Alignment

Bidisha Sharma, Haizhou Li

Abstract

Speech-to-singing (S2S) conversion requires precise temporal alignment between spoken lyrics and target singing vocals. This paper presents a hybrid alignment strategy combining model-based forced alignment with feature-based Dynamic Time Warping (DTW). By leveraging reliable speech word boundaries as anchors and refining singing boundaries through a localized search, the proposed method achieves a 47.5% reduction in word boundary error, significantly enhancing the quality of synthesized singing.

Introduction

In speech-to-singing (S2S) conversion, the objective is to transform a spoken recording into a singing version that adheres to a specific melody and rhythm. A fundamental challenge in this process is achieving accurate temporal alignment—mapping specific segments of speech to their corresponding segments in the singing audio.

This task is complicated by significant acoustic and temporal differences between the two modalities. Singing often involves non-linear temporal stretching (e.g., elongated vowels), spectral shifts due to physiological changes like a lowered larynx, and distinct prosodic features such as vibrato and wide pitch ranges. Traditional automated methods, such as pure DTW, often suffer from error accumulation over long sequences, while manual alignment is prohibitively slow.

Proposed Methodology

The authors introduce a multi-stage pipeline that integrates model-based and feature-based techniques to ensure robust alignment.

The proposed multi-stage alignment pipeline, integrating model-based and feature-based components.

1. Model-Based Forced Alignment

The process begins with initial word boundary estimation. Spoken lyrics are aligned using the Montreal Forced Aligner (MFA). Simultaneously, singing vocals are aligned using a singing-adapted acoustic model. The core hypothesis is that speech acoustic models provide more reliable "anchor" boundaries than singing models.

2. Singing Word Boundary Correction

To address the lower precision of singing models, a "Window Shifter" component refines the initial singing word boundaries. Using the speech word as a reference, the system evaluates potential singing word boundaries within a ±100ms neighborhood. The optimal shift is determined by minimizing the DTW cost between speech and singing features.

3. Feature-Based Fine Alignment

Following word-level correction, the system performs frame-level alignment. By executing DTW using 39-dimensional Mel-frequency cepstral coefficients (MFCCs) strictly within the established word boundaries, the algorithm prevents the "drift" typically associated with long-sequence feature-based alignment.

Frame-level mapping between singing and speech words within established boundaries.

Experimental Setup and Datasets

The study utilized a parallel speech-singing corpus from the National University of Singapore's HLT Laboratory. The dataset comprises 80 English songs performed by 8 singers (4 male, 4 female), totaling 2,314 utterances. This represents approximately 116 minutes of speech and 237 minutes of singing. Manual annotations of word boundaries served as the ground truth for evaluation.

Key Results and Discussion

The hybrid approach was evaluated against a standard DTW baseline and a basic forced alignment method.

  • Quantitative Accuracy: The Mean Word Boundary Error (WBE) was reduced from 315.50 ms in the baseline to 165.35 ms, representing a 47.5% improvement in alignment precision.
  • Subjective Quality: In listening tests, the synthesized singing produced using the proposed alignment achieved a Mean Opinion Score (MOS) of 4.33, compared to 2.89 for the baseline.
  • User Preference: Listeners preferred the proposed method in 87.08% of head-to-head comparisons.

The success of this method stems from treating speech boundaries as reliable landmarks and employing a "divide and conquer" strategy that restricts DTW to localized word windows, thereby avoiding global optimization errors.

Conclusion and Future Work

The combination of model-based anchors and feature-based refinement provides a significant leap in automated S2S alignment. While highly effective, the authors note that local word-by-word optimization may not always yield the globally optimal path for an entire sentence. Future research will explore global strategies for boundary correction and improved handling of stylistic musical silences and breaths.

Summary updated

Abstract

Speech and singing are different in many ways. In this work, we propose a novel method to align phonetically identical spoken lyric with a singing vocal in a speech-singing parallel corpus, that is needed in speech-to-singing conversion. We attempt to align speech to singing vocal using a combination of model-based forced alignment and feature-based dynamic time warping (DTW). We first obtain the word boundaries of speech and singing vocals with forced alignment using speech and singing adapted acoustic models, respectively. We consider that speech acoustic models are more accurate than singing acoustic models, therefore, boundaries of spoken words are more accurate than sung words. By searching in the neighborhood of the sung word boundaries in the singing vocal, we hope to improve the alignment between spoken words and sung words. Considering the word boundaries as landmark, we perform speech-to-singing alignment at frame-level using DTW. The proposed method is able to achieve a 47.5% reduction in terms of word boundary error over the baseline, and subsequent improvement of singing quality in a speech-to-singing conversion system.
BibTeX
@inproceedings{sharma2019combination,
  author = {Bidisha Sharma and Haizhou Li},
  title = {{A Combination of Model-Based and Feature-Based Strategy for Speech-to-Singing Alignment}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-1942},
}
Figure