Music-Robust Automatic Lyrics Transcription Of Polyphonic Music

Xiaoxue Gao, Chitralekha Gupta, Haizhou LiZenodo 2022 · DOI · OpenAlex

Abstract

Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription, we propose a strategy of combining music-removed features (singing vocals) and music-present features (original mix). We show that these two sets of features complement each other, improving the robustness of the acoustic model. Furthermore, language model interpolation between a general-purpose and an in-domain lyrics-specific model provides further gains. Our experiments show that our proposed strategy outperforms existing systems, especially in genres like metal where background music is dominant.

Introduction

Automatic Speech Recognition (ASR) has seen significant success in clean environments, but Automatic Lyrics Transcription (ALT) remains a difficult task. In polyphonic music, the singing voice is often obscured by loud instruments. Previous attempts typically relied on either raw audio (where music drowns out vocals) or source separation (which introduces digital artifacts). This research addresses these limitations by leveraging both raw and separated audio streams to create a more robust transcription system.

Proposed Methodology

The core of the proposed framework is the extraction of "Music-Robust Features." Instead of choosing between the original polyphonic mix and the extracted vocal track, the system concatenates MFCCs from both. This allows the model to benefit from the clarity of the separated vocals while using the raw audio as a reference to compensate for separation artifacts.

Figure 4: Overview of the proposed music-robust AM training and lyrics transcription inference pipeline.

The architecture utilizes a Factorized Time-Delay Neural Network (TDNN-F) for acoustic modeling. To further refine the output, the researchers employ Language Model (LM) interpolation. By blending a general English model with one specifically trained on song lyrics, the system better captures the unique linguistic structures found in music.

Datasets and Evaluation

The models were trained on the DALI and NUS datasets, providing a broad range of English songs. For evaluation, the team used established benchmarks like Hansen, Mauch, and Jamendo, providing manually segmented 20-30 second chunks to ensure high-quality ground truth for the research community.

Figure 6: Summary of the datasets used for training and evaluation, including song counts and total duration.

Key Results and Insights

The hybrid feature approach proved highly effective, yielding a 5-6% improvement in Word Error Rate (WER) over raw-audio baselines and outperforming MIREX 2020 state-of-the-art systems by 8-9%.

A significant finding was the system's performance in the "Metal" genre. In tracks where instruments are traditionally overwhelming, the music-robust features allowed the AI to "ignore" the background noise more effectively than standard models, reducing WER from 60.64% to 56.60%. This confirms that the raw audio provides necessary context for the model to distinguish between vocal signals and instrumental interference.

Conclusion and Future Directions

While the system excels in noisy environments, it faced challenges with hip-hop, likely due to the high vocal density and speed causing distortions during the separation phase. Future research will explore end-to-end integration of the vocal extractor and transcriber to minimize these artifacts and further improve accuracy across all musical genres.

Summary updated

Abstract

Lyrics transcription of polyphonic music is challenging because singing vocals are corrupted by the background music. To improve the robustness of lyrics transcription to the background music, we propose a strategy of combining the features that emphasize the singing vocals, i.e. music-removed features that represent singing vocal extracted features, and the features that capture the singing vocals as well as the background music, i.e. music-present features. We show that these two sets of features complement each other, and their combination performs better than when they are used alone, thus improving the robustness of the acoustic model to the background music. Furthermore, language model interpolation between a general-purpose language model and an in-domain lyrics-specific language model provides further improvement in transcription results. Our experiments show that our proposed strategy outperforms the existing lyrics transcription systems for polyphonic music. Moreover, we find that our proposed music-robust features specially improve the lyrics transcription performance in metal genre of songs, where the background music is loud and dominant.
BibTeX
@inproceedings{gao2022music,
  author = {Xiaoxue Gao and Chitralekha Gupta and Haizhou Li},
  title = {{Music-Robust Automatic Lyrics Transcription Of Polyphonic Music}},
  booktitle = {Zenodo},
  year = {2022},
  doi = {10.5281/zenodo.6798107},
}

Loading related content...

Figure