KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms

Chien-Feng Liao, Jen-Yu Liu, Yi-Hsuan Yang

Abstract

In this paper, we propose a novel neural network model called KaraSinger for a less-studied singing voice synthesis (SVS) task named score-free SVS, in which the prosody and melody are spontaneously decided by machine. KaraSinger comprises a vector-quantized variational autoencoder (VQ-VAE) that compresses the Mel-spectrograms of singing audio to sequences of discrete codes, and a language model (LM) that learns to predict the discrete codes given the corresponding lyrics. For the VQ-VAE part, we employ a Connectionist Temporal Classification (CTC) loss to encourage the discrete codes to carry phoneme-related information. For the LM part, we use location-sensitive attention for learning a robust alignment between the input phoneme sequence and the output discrete code. We keep the architecture of both the VQ-VAE and LM light-weight for fast training and inference speed. We validate the effectiveness of the proposed design choices using a proprietary collection of 550 English pop songs sung by multiple amateur singers. The result of a listening test shows that KaraSinger achieves high scores in intelligibility, musicality, and the overall quality.

Introduction

Traditional Singing Voice Synthesis (SVS) systems are demanding, requiring both lyrics and a musical score (MIDI-like data specifying pitch and duration) as input. While most users can easily write lyrics, composing a coherent melody remains a significant barrier. Existing "score-free" models like OpenAI’s Jukebox can generate singing from lyrics but face hurdles in computational cost and alignment issues, often suffering from "word-skipping."

KaraSinger aims to provide a "lite," efficient alternative that learns to spontaneously decide on prosody and melody based on training data, making AI-driven singing accessible to non-musicians. By combining a hierarchical VQ-VAE with a lightweight Transformer-based language model, the system can be trained on a single consumer GPU while maintaining high intelligibility.

Methodology

The authors employ a two-stage architecture optimized for efficiency and alignment.

Hierarchical VQ-VAE over Mel-spectrograms

Instead of working with raw audio waveforms, KaraSinger operates on Mel-spectrograms to reduce computational complexity. A Vector Quantized Variational Autoencoder (VQ-VAE) compresses the spectrograms into three levels of discrete "audio tokens": Top (coarse/content), Middle, and Bottom (fine/style).

The hierarchical VQ-VAE architecture of KaraSinger, highlighting the integration of CTC loss for phonetic alignment.

To ensure the discrete tokens represent the lyrics, a Connectionist Temporal Classification (CTC) loss is added to the top-level encoder. This forces the model to align its latent codes with phonemes, which is crucial for maintaining intelligibility in low-data regimes.

Language Model (LM)

Once the audio is tokenized, a language model learns to predict these tokens from lyrics. The Lyrics Encoder uses a seq2seq architecture with location-sensitive attention to map phonemes to top-level codes. Instead of modeling the middle and bottom levels separately, KaraSinger interleaves them into a single sequence. A Linear Transformer then predicts these style tokens, conditioned on the top-level content, using "tick embeddings" to track interleaved sequence positions.

Experimental Setup

The researchers curated a proprietary dataset of 550 English pop songs performed by various amateur singers, totaling roughly 20 hours of audio. This is a remarkably small dataset compared to the 1.2 million songs used by models like Jukebox. By moving away from raw audio to Mel-spectrograms, training time was reduced from weeks on a cluster to just four days on a single NVIDIA RTX 2080Ti.

Results

Subjective listening tests compared KaraSinger to two ablated versions: one without CTC loss ("noCTC") and one using a standard 3-level hierarchical decoder.

Subjective evaluation results comparing KaraSinger (Proposed) against baseline models in terms of intelligibility, musicality, and overall quality.

As shown in the results, KaraSinger achieved an intelligibility score of 4.23/5, significantly higher than the version without CTC (1.30/5), proving that the CTC loss is essential for lyric alignment. It also outperformed the Jukebox-style hierarchy in both musicality (3.85) and overall quality (3.67). Additionally, the model achieves a 1:1 inference speed, allowing for real-time generation.

Conclusion and Limitations

KaraSinger demonstrates that high-quality, score-free SVS is achievable with limited data and hardware. While the model produces natural-sounding vibrato, it faces challenges with pitch stability and maintaining long-term melodic structure over full songs. Future work will explore conditioning the model on instrumental accompaniments to allow for more integrated musical generation.

Reference: Liao, C. F., Liu, J. Y., & Yang, Y. H. (2021). KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms. arXiv:2110.04005.

Summary updated

Abstract

In this paper, we propose a novel neural network model called KaraSinger for a less-studied singing voice synthesis (SVS) task named score-free SVS, in which the prosody and melody are spontaneously decided by machine. KaraSinger comprises a vector-quantized variational autoencoder (VQ-VAE) that compresses the Mel-spectrograms of singing audio to sequences of discrete codes, and a language model (LM) that learns to predict the discrete codes given the corresponding lyrics. For the VQ-VAE part, we employ a Connectionist Temporal Classification (CTC) loss to encourage the discrete codes to carry phoneme-related information. For the LM part, we use location-sensitive attention for learning a robust alignment between the input phoneme sequence and the output discrete code. We keep the architecture of both the VQ-VAE and LM light-weight for fast training and inference speed. We validate the effectiveness of the proposed design choices using a proprietary collection of 550 English pop songs sung by multiple amateur singers. The result of a listening test shows that KaraSinger achieves high scores in intelligibility, musicality, and the overall quality.
BibTeX
@inproceedings{liao2022karasinger,
  author = {Chien-Feng Liao and Jen-Yu Liu and Yi-Hsuan Yang},
  title = {{KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=nhSMrV6hro},
}
Figure