TL;DR
This paper introduces a novel, end-to-end system for aligning audio with lyrics across multiple languages. By employing contrastive learning to map audio and text into a shared embedding space, the system achieves state-of-the-art accuracy on English lyrics alignment and demonstrates robust performance on other languages, even with limited training data.
What Problem Does This Paper Address?
Automatically aligning lyrics to music, a task crucial for applications like karaoke, lyric display, and music information retrieval, has historically been challenging. Existing methods often rely on complex speech recognition toolkits or end-to-end systems using Connectionist Temporal Classification (CTC). While toolkits offer robustness, they are difficult to train and require expert knowledge. CTC-based systems, though simpler, struggle with the inherent inaccuracies and unannotated sections common in lyrics, leading to limited alignment precision. This paper aims to bridge this gap by developing a simpler, yet more accurate and versatile system for audio-lyrics alignment, particularly for multilingual scenarios.
Methods
The core of the proposed system is a contrastive learning approach that learns to map audio and lyrics into a shared embedding space.
- Audio Encoder: A residual network processes spectrogram representations of the audio to extract local phonetic content. This encoder is designed to capture the singing voice's characteristics within short time windows.
- Text Encoder: This component processes lyrics, which can be represented as characters or phonemes. Crucially, it incorporates a context window, allowing it to consider neighboring symbols when generating embeddings. This contextualization helps in more accurately representing pronunciation, which is influenced by surrounding text. For multilingual support, a language-conditioning mechanism is added.
- Contrastive Learning: Instead of relying on precise symbol-level annotations, the system uses weakly labeled data. It trains by maximizing the similarity between corresponding audio and lyric embeddings (positive pairs) while minimizing similarity for non-corresponding pairs (negative pairs). This approach avoids the need for an explicit "epsilon" symbol or fixed state space, which are limitations of CTC.
- Line-Based Decoding: To improve alignment accuracy and prevent misplacements of lyrics at line boundaries, a line-masking strategy is employed. This mask guides the decoding process to keep all tokens of a lyrical line clustered together within an estimated line interval.
Datasets
- Training: The system is trained on a large dataset of approximately 400 hours of music, sampled from a collection of 87,785 songs. This dataset includes professional-quality recordings in English, Spanish, German, and French.
- Evaluation: The primary evaluation is conducted on the JamendoLyrics Multi-Lang dataset, an extension of the JamendoLyrics dataset. This dataset features Creative Commons songs and includes English, Spanish, German, and French. The authors also introduce manually annotated word-level timings for this dataset to enable robust evaluation of multilingual alignment.
Key Insights and Ideas
- Contrastive Learning for Alignment: The paper demonstrates that contrastive learning is a powerful and flexible alternative to CTC for audio-lyrics alignment, especially when dealing with weakly labeled data.
- Contextual Text Embeddings: Representing text symbols (especially characters) with contextual information significantly improves alignment accuracy, making it competitive with phoneme-based approaches. This suggests that data-driven contextual embeddings can implicitly learn phonetic nuances.
- Multilingual Robustness: The proposed system, particularly with character-based contextual embeddings and language conditioning, shows remarkable robustness across different languages, outperforming previous multilingual approaches.
- Line-Based Decoding Simplifies Constraints: The introduced line-based decoding mechanism effectively improves alignment by leveraging lyrical structure, without requiring external vocal activity or boundary detection modules.
Key Results
- The system achieves an average absolute error (AAE) below 0.2 seconds on the JamendoLyrics dataset, setting a new state-of-the-art for English lyrics alignment.
- The best performing model (M6, character-based, contextual, no line-based decoding) achieved an AAE of 0.15s and a PCO of 92% on the English JamendoLyrics dataset.
- With language conditioning, the system achieved an AAE of 0.18s on the JamendoLyrics Multi-Lang dataset, outperforming previous multilingual approaches and setting a new benchmark.
- The system requires fewer dependencies compared to previous state-of-the-art methods, making it simpler and more adaptable.
Limitations and Future Work
- Limitations: While the system performs well, the paper notes that the performance on non-English languages can be affected by the quality of the language conditioning. The authors also mention that their approach might not be ideal for highly overlapping vocals (e.g., duets) without modifications to the decoding stage.
- Future Work: The authors suggest exploring the decoding stage to handle overlapping vocals and duets. They also propose investigating the system's potential for low-resource languages through transfer and few-shot learning, leveraging the reusable nature of the embedding-based architecture. Further research into more complex text encoder architectures is also mentioned as a possibility for improvement.
Summary updated