NUS Speak-to-Sing: A Web Platform for Personalized Speech-to-Singing Conversion

Chitralekha Gupta, Karthika Vijayan, Bidisha Sharma, Xiaoxue Gao, Haizhou Li

Abstract

The NUS Speak-to-Sing platform enables users to generate high-quality singing vocals in their own voice by simply reading lyrics aloud. The system bridges the gap between speech and singing by combining the user's vocal identity with the professional prosody of a trained singer. This is achieved through a template-based conversion scheme utilizing deep learning-based spectral mapping and robust temporal alignment.

Introduction

While many individuals enjoy singing, few possess the formal training required for professional-level pitch control and expression. Traditional solutions like pitch correction (Auto-Tune) or artificial synthesizers (Vocaloid) either fail to handle flat speech or lack personalization. The Speech-to-Singing (STS) task requires two main components: aligning spoken lyrics to a melody and transforming speech spectra into the rich, harmonic-heavy spectra characteristic of professional singing while preserving the user's identity.

System Architecture

The NUS Speak-to-Sing system follows a template-based conversion pipeline. The architecture integrates user input with professional templates to synthesize a final singing performance.

Overview of the NUS Speak-to-Sing system architecture, illustrating the flow from user speech and singing templates to the final synthesized output.

As shown in the figure above, the process involves several critical modules:

  1. Template Preparation: A database of professional studio recordings provides the "gold standard" for pitch (F0) and timing.
  2. Temporal Alignment: Dynamic Time Warping (DTW) is used to sync user speech with the template. The system employs "tandem features"—a mix of acoustic features—to ensure robust alignment despite the differences between speech and singing.
  3. Deep-Spectral Mapping: A deep learning model maps the user's spoken spectral envelope to a "singing version." This adds singing-specific traits, such as the "singer’s formant," while maintaining the user's recognizable voice.
  4. Synthesis: A vocoder combines the transformed spectra with the professional template's prosody to produce the final output.

Key Innovations

Identity vs. Technique

The core philosophy of the system is the separation of vocal identity (spectral characteristics) from musical technique (prosody). By sourcing identity from the user and technique from a professional template, the system bypasses the user's lack of vocal training.

Deep-Spectral Mapping

Spoken speech typically lacks the harmonic richness and modulation found in singing. The deep-spectral map actively transforms the spectral shape to mimic the physiological behavior of a trained singer's vocal tract, ensuring the output sounds like a professional performance rather than just "tuned" speech.

Results and Implementation

The authors implemented this technology as an interactive web platform. Users can select a song from a library and record themselves reading the lyrics.

  • Quality: The system produces enhanced singing vocals compared to earlier iterations like I²R Speech2Singing.
  • Personalization: The deep-spectral map successfully retains the user's unique vocal "fingerprint," allowing them to "sing" notes beyond their natural range.
  • Accessibility: Designed for real-world environments, the system is robust enough for home recordings.

Conclusion and Future Work

NUS Speak-to-Sing represents a significant step toward making professional-quality singing accessible to everyone. Future work aims to reduce the dependency on manually labeled templates and expand the system's flexibility to support arbitrary song uploads and cross-gender transformations.

Summary updated

BibTeX
@inproceedings{gupta2019nus,
  author = {Chitralekha Gupta and Karthika Vijayan and Bidisha Sharma and Xiaoxue Gao and Haizhou Li},
  title = {{NUS Speak-to-Sing: A Web Platform for Personalized Speech-to-Singing Conversion}},
  booktitle = {Interspeech},
  year = {2019},
}
Figure