TL;DR
BARS (Beat-Adaptive Rap Synthesis) is a non-autoregressive generative model that synthesizes rapping vocals from lyrics, a beat track, and a short reference voice. By leveraging Flow Matching and a novel asymmetric positional embedding mechanism, it achieves high-fidelity, rhythmically coherent rap with independent control over tempo and zero-shot voice cloning.
What Problem Does This Paper Address?
Synthesizing rap is significantly harder than standard Text-to-Speech (TTS) or even singing voice synthesis. Rap is defined by its "flow"—a complex interplay of dynamic stress patterns, syncopation, and deliberate timing deviations that must align with an underlying beat while maintaining lyric integrity.
Existing systems often fall into two traps:
- Rigidity: They require predefined pitch and duration templates, which are incompatible with the irregular nature of rap.
- Autoregressive Limitations: Models like Freestyler are slow to decode and tie the rapping pace strictly to the beat duration. This prevents "creative" pacing—like rapping a fast verse over a slow beat—and often leads to alignment errors (skipping or repeating words) if the beat is too long or short.
BARS aims to provide a flexible, fast, and robust tool for artists to prototype vocal deliveries with full control over pacing and style.
Methods
The BARS architecture is built on three main pillars:
- Flow Matching: A non-autoregressive generative framework that is faster and often more stable than traditional diffusion models. It learns to transform Gaussian noise into audio latents conditioned on the beat and lyrics.
- ByT5 + Transformer Adapter: Instead of using speech-pretrained encoders (which often struggle with rap's heavy use of slang, abbreviations, and numbers), the authors use a byte-level ByT5 encoder. They add a lightweight Transformer adapter to bridge the modality gap between text embeddings and audio latents.
- Asymmetric Scaled RoPE (A-RoPE): This is the paper's primary innovation for solving alignment. In standard cross-attention, a strong beat signal can "distract" the model, causing it to lose its place in the lyrics. A-RoPE applies standard Rotary Position Embeddings (RoPE) to the audio query but scales the lyric key indices by the ratio of audio length to lyric length ($N/L$). This effectively "stretches" the lyrics across the timeline of the beat, providing a strong sequential prior.
The model also uses span masking during training, which allows it to perform "infilling"—editing specific words or phrases in a generated track while keeping the surrounding vocal style and rhythm intact.
Datasets
The model was trained on approximately 1,500 hours of English rap music featuring a diverse range of subgenres and both male and female artists. Each training sample included the rap vocal, corresponding lyrics, the beat track, and a 3-second reference clip for speaker identity. The data was split 90/5/5 for training, validation, and testing, ensuring no rapper overlap between sets.
Key Insights and Ideas
- The Alignment "Aha" Moment: The authors realized that without positional guidance, the model prioritizes rhythmic alignment with the beat at the expense of lyric accuracy. A-RoPE fixes this by creating a "soft" diagonal constraint in the attention map, ensuring the model moves through the lyrics at a pace proportional to the audio length.
- Adapter vs. Pre-training: Interestingly, a lightweight adapter on top of a text-only model (ByT5) performed as well as or better than heavy speech-pretrained encoders like SpeechT5. This suggests that for specialized domains like rap, byte-level flexibility is more valuable than general speech priors.
- Immediate Lyric Utilization: Training curves show that models with A-RoPE start utilizing lyric information almost immediately, whereas models without it have a "dead zone" of ~50k steps where they ignore lyrics entirely before suddenly "snapping" into alignment.
Key Results
- High Intelligibility: BARS achieved a Word Error Rate (WER) of 5.1% on matched lyric-beat pairs, which is effectively the same as the error rate for human-recorded ground truth (due to the inherent difficulty of transcribing rap).
- Inference Speed: The model is highly efficient, with a Real-Time Factor (RTF) of 0.027. It can generate 10 seconds of high-quality audio in just 0.27 seconds on a single RTX 4090.
- Tempo Robustness: In "mismatched" tests (where lyrics were intentionally rapped faster or slower than the original beat), BARS maintained significantly lower WER than variants without A-RoPE, proving it can adapt its pacing creatively.
- Zero-Shot Quality: Speaker similarity scores (0.80 cosine similarity) matched the ground truth upper bound, demonstrating excellent voice cloning from just a 3-second sample.
Limitations and Future Work
The primary limitation acknowledged is that the study is restricted to English. While the ByT5 encoder is byte-level and theoretically language-agnostic, different languages have different byte-per-character ratios. This could affect the $N/L$ scaling factor in A-RoPE. The authors have released preliminary multilingual examples on their demo page but state that systematic evaluation and adaptation for other languages are left for future work. Additionally, while the model supports up to 60-second clips, performance on extremely long-form generation hasn't been fully explored.
Summary updated