Abstract
In this paper, we propose a novel neural network model called KaraSinger for a less-studied singing voice synthesis (SVS) task named score-free SVS, in which the prosody and melody are spontaneously decided by machine. KaraSinger comprises a vector-quantized variational autoencoder (VQ-VAE) that compresses the Mel-spectrograms of singing audio to sequences of discrete codes, and a language model (LM) that learns to predict the discrete codes given the corresponding lyrics. For the VQ-VAE part, we employ a Connectionist Temporal Classification (CTC) loss to encourage the discrete codes to carry phoneme-related information. For the LM part, we use location-sensitive attention for learning a robust alignment between the input phoneme sequence and the output discrete code. We keep the architecture of both the VQ-VAE and LM light-weight for fast training and inference speed. We validate the effectiveness of the proposed design choices using a proprietary collection of 550 English pop songs sung by multiple amateur singers. The result of a listening test shows that KaraSinger achieves high scores in intelligibility, musicality, and the overall quality.
Introduction
Traditional Singing Voice Synthesis (SVS) systems are demanding, requiring both lyrics and a musical score (MIDI-like data specifying pitch and duration) as input. While most users can easily write lyrics, composing a coherent melody remains a significant barrier. Existing "score-free" models like OpenAI’s Jukebox can generate singing from lyrics but face hurdles in computational cost and alignment issues, often suffering from "word-skipping."
KaraSinger aims to provide a "lite," efficient alternative that learns to spontaneously decide on prosody and melody based on training data, making AI-driven singing accessible to non-musicians. By combining a hierarchical VQ-VAE with a lightweight Transformer-based language model, the system can be trained on a single consumer GPU while maintaining high intelligibility.
Methodology
The authors employ a two-stage architecture optimized for efficiency and alignment.
Hierarchical VQ-VAE over Mel-spectrograms
Instead of working with raw audio waveforms, KaraSinger operates on Mel-spectrograms to reduce computational complexity. A Vector Quantized Variational Autoencoder (VQ-VAE) compresses the spectrograms into three levels of discrete "audio tokens": Top (coarse/content), Middle, and Bottom (fine/style).
To ensure the discrete tokens represent the lyrics, a Connectionist Temporal Classification (CTC) loss is added to the top-level encoder. This forces the model to align its latent codes with phonemes, which is crucial for maintaining intelligibility in low-data regimes.
Language Model (LM)
Once the audio is tokenized, a language model learns to predict these tokens from lyrics. The Lyrics Encoder uses a seq2seq architecture with location-sensitive attention to map phonemes to top-level codes. Instead of modeling the middle and bottom levels separately, KaraSinger interleaves them into a single sequence. A Linear Transformer then predicts these style tokens, conditioned on the top-level content, using "tick embeddings" to track interleaved sequence positions.
Experimental Setup
The researchers curated a proprietary dataset of 550 English pop songs performed by various amateur singers, totaling roughly 20 hours of audio. This is a remarkably small dataset compared to the 1.2 million songs used by models like Jukebox. By moving away from raw audio to Mel-spectrograms, training time was reduced from weeks on a cluster to just four days on a single NVIDIA RTX 2080Ti.
Results
Subjective listening tests compared KaraSinger to two ablated versions: one without CTC loss ("noCTC") and one using a standard 3-level hierarchical decoder.
As shown in the results, KaraSinger achieved an intelligibility score of 4.23/5, significantly higher than the version without CTC (1.30/5), proving that the CTC loss is essential for lyric alignment. It also outperformed the Jukebox-style hierarchy in both musicality (3.85) and overall quality (3.67). Additionally, the model achieves a 1:1 inference speed, allowing for real-time generation.
Conclusion and Limitations
KaraSinger demonstrates that high-quality, score-free SVS is achievable with limited data and hardware. While the model produces natural-sounding vibrato, it faces challenges with pitch stability and maintaining long-term melodic structure over full songs. Future work will explore conditioning the model on instrumental accompaniments to allow for more integrated musical generation.
Reference: Liao, C. F., Liu, J. Y., & Yang, Y. H. (2021). KaraSinger: Score-Free Singing Voice Synthesis with VQ-VAE using Mel-spectrograms. arXiv:2110.04005.
Summary updated