TL;DR
This paper introduces a novel end-to-end deep learning model for aligning lyrics to polyphonic music, achieving state-of-the-art performance with significantly simpler training requirements. By leveraging a modified Wave-U-Net architecture and Connectionist Temporal Classification (CTC) loss, the system can learn directly from raw audio and weak, line-level lyric annotations, overcoming limitations of previous methods.
What Problem Does This Paper Address?
Aligning lyrics to music, a task crucial for applications like karaoke, song retrieval, and content moderation, is surprisingly challenging. While text-to-speech alignment is relatively mature, music presents unique difficulties: multiple, often correlated, sound sources (polyphony) violate independence assumptions of traditional speech systems, and singing voices exhibit greater variability in pitch, timbre, and tempo than speech. Even with pre-existing lyrics, precisely synchronizing them to the audio remains a hard problem, with existing state-of-the-art systems still exhibiting significant alignment errors. Many prior approaches relied on complex multi-module pipelines, fine-grained annotations, or were limited to solo vocal recordings, failing to generalize to real-world polyphonic music.
Methods
The core of the proposed system is an end-to-end acoustic model based on a modified Wave-U-Net architecture. This model directly predicts character probabilities from raw audio, learning multi-scale representations of the signal components. Unlike previous methods that often relied on separate vocal separation or feature extraction modules, this approach integrates these capabilities within a single, jointly optimized network.
Key methodological innovations include:
- Modified Wave-U-Net: The architecture uses a series of downsampling and upsampling blocks to capture features at multiple time resolutions, enabling it to model both low-level audio details and higher-level musical structures.
- End-to-End Training: The entire model is trained jointly, avoiding the error propagation issues common in multi-stage pipelines.
- Connectionist Temporal Classification (CTC) Loss: This loss function is applied to weak, line-level lyric annotations. CTC allows for flexible alignment by marginalizing over all possible alignment paths between the audio and the target lyrics, rather than relying on hard Viterbi alignments which can bias training. This significantly simplifies the annotation requirement.
- Convolutional Architecture: The model uses purely convolutional layers, eschewing recurrent connections. This choice is motivated by findings suggesting that RNNs can sometimes lead to lower temporal accuracy in alignment tasks when used with CTC loss.
Datasets
- Internal Training Dataset: Comprised of 44,232 songs with English lyrics across various genres (Pop, Rock, Hip-Hop). This dataset includes annotations for the start and end times of each lyrical line.
- Mauch Dataset: Used for evaluation, this dataset is known to contain only Pop music and was previously used in the MIREX 2017 lyrics alignment challenge.
- Jamendo Dataset: A newly introduced, freely available dataset of 20 songs from nine genres. This dataset aims to provide a more diverse and comparable benchmark for evaluating alignment and transcription systems, addressing the bias towards Pop music in existing datasets.
Key Insights and Ideas
- End-to-end learning simplifies alignment: By training a single model, the system avoids the complexities and error propagation of multi-module pipelines, leading to better overall performance.
- Weak annotations are sufficient: The use of CTC loss with line-level annotations dramatically reduces the annotation burden, making it feasible to train on large, real-world datasets.
- Phoneme-agnostic approach is beneficial for singing: Directly predicting characters avoids the need for pronunciation dictionaries, which are often ill-suited for the varied pronunciation and non-lexical vocalizations found in singing.
- Convolutional networks can be effective for CTC alignment: The paper demonstrates that a purely convolutional architecture can achieve strong alignment results without the potential drawbacks of RNNs in this specific context.
Key Results
- State-of-the-art alignment accuracy: The system achieved a mean alignment error (AE) of 0.35s on the standard Mauch dataset, outperforming previous MIREX 2017 systems by an order of magnitude. On the more diverse Jamendo dataset, the AE was 0.82s.
- High percentage of correct word predictions: The system achieved 77.2% correct word predictions on the Mauch dataset, a significant improvement over prior methods.
- Useful transcription capabilities: While not primarily designed for transcription, the model's output, when combined with a language model, yielded competitive word error rates (WER), suggesting its potential for retrieval tasks.
- Impact of vocal separation: Preliminary experiments showed that further improving alignment accuracy by using a vocal separation pre-processing step, suggesting potential for multi-task learning.
Limitations and Future Work
- Phoneme-based models for other languages: While the current character-based approach is effective for English, phoneme-based models might be more adaptable to other languages, requiring only a replacement of the phoneme dictionary.
- Potential for multi-task learning: The authors suggest that future work could explore multi-task learning to unify objectives for alignment, transcription, and potentially vocal separation, leveraging larger datasets.
- Impact of vocal separation on performance: While using a vocal separator improved results, the separator itself was not optimized for lyrics alignment, indicating room for further gains.
- Transcription accuracy: The character error rate (CER) was around 50%, and WER was considerably higher, indicating that while alignment is strong, precise transcription still presents challenges, especially for misspelt words.
Summary updated