Abstract
Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription, we propose a strategy of combining music-removed features (singing vocals) and music-present features (original mix). We show that these two sets of features complement each other, improving the robustness of the acoustic model. Furthermore, language model interpolation between a general-purpose and an in-domain lyrics-specific model provides further gains. Our experiments show that our proposed strategy outperforms existing systems, especially in genres like metal where background music is dominant.
Introduction
Automatic Speech Recognition (ASR) has seen significant success in clean environments, but Automatic Lyrics Transcription (ALT) remains a difficult task. In polyphonic music, the singing voice is often obscured by loud instruments. Previous attempts typically relied on either raw audio (where music drowns out vocals) or source separation (which introduces digital artifacts). This research addresses these limitations by leveraging both raw and separated audio streams to create a more robust transcription system.
Proposed Methodology
The core of the proposed framework is the extraction of "Music-Robust Features." Instead of choosing between the original polyphonic mix and the extracted vocal track, the system concatenates MFCCs from both. This allows the model to benefit from the clarity of the separated vocals while using the raw audio as a reference to compensate for separation artifacts.
The architecture utilizes a Factorized Time-Delay Neural Network (TDNN-F) for acoustic modeling. To further refine the output, the researchers employ Language Model (LM) interpolation. By blending a general English model with one specifically trained on song lyrics, the system better captures the unique linguistic structures found in music.
Datasets and Evaluation
The models were trained on the DALI and NUS datasets, providing a broad range of English songs. For evaluation, the team used established benchmarks like Hansen, Mauch, and Jamendo, providing manually segmented 20-30 second chunks to ensure high-quality ground truth for the research community.
Key Results and Insights
The hybrid feature approach proved highly effective, yielding a 5-6% improvement in Word Error Rate (WER) over raw-audio baselines and outperforming MIREX 2020 state-of-the-art systems by 8-9%.
A significant finding was the system's performance in the "Metal" genre. In tracks where instruments are traditionally overwhelming, the music-robust features allowed the AI to "ignore" the background noise more effectively than standard models, reducing WER from 60.64% to 56.60%. This confirms that the raw audio provides necessary context for the model to distinguish between vocal signals and instrumental interference.
Conclusion and Future Directions
While the system excels in noisy environments, it faced challenges with hip-hop, likely due to the high vocal density and speed causing distortions during the separation phase. Future research will explore end-to-end integration of the vocal extractor and transcriber to minimize these artifacts and further improve accuracy across all musical genres.
Summary updated