Singing Language Identification using a Deep Phonotactic Approach

Lenny Renault, Andrea Vaglio, Romain HennequinICASSP 2021 · DOI · OpenReview

Abstract: Extensive works have tackled Language Identification (LID) in the speech domain, however their application to the singing voice trails and performances on Singing Language Identification (SLID) can be improved leveraging recent progresses made in other singing related tasks. This work presents a modernized phonotactic system for SLID on polyphonic music: phoneme recognition is performed with a Connectionist Temporal Classification (CTC)-based acoustic model trained with multilingual data, before language classification with a recurrent model based on the phonemes estimation. The full pipeline is trained and evaluated with a large and publicly available dataset, with unprecedented performances. First results of SLID with out-of-set languages are also presented.

Introduction

Identifying the language of a song, known as Singing Language Identification (SLID), is a foundational task for organizing massive music catalogs. While Language Identification (LID) is well-established for speech, singing introduces unique hurdles. Prosodic variability—such as pitch, vibrato, and elongated vowels—distorts the natural "sound" of a language. Furthermore, background instruments often overlap with and mask vocal frequencies, and large-scale datasets of aligned lyrics and audio are rare.

Traditional methods have relied on metadata, which is frequently missing or inaccurate, or "i-vectors," which are statistical summaries of audio features. This research moves toward a phonotactic approach, identifying specific sequences of phonemes (the smallest units of sound) that characterize a language.

A Modernized Phonotactic Architecture

The proposed system utilizes a two-stage deep learning pipeline designed to "read" phonemes from singing voices and use their sequences to identify the language.

Proposed SLID system architecture featuring a CTC acoustic model and a recurrent language classifier.
  1. Acoustic Model ($F$): A Convolutional Recurrent Neural Network (CRNN) trained using Connectionist Temporal Classification (CTC). CTC allows the model to learn from "weakly aligned" data, outputting a phoneme posteriorgram—a grid showing the probability of different International Phonetic Alphabet (IPA) symbols over time.
  2. Language Classifier ($G$): A bidirectional Long Short-Term Memory (LSTM) network that processes the phoneme posteriorgram. It learns the "grammar" of sounds, identifying phoneme transitions characteristic of specific languages.

A key innovation is the use of Joint Training, where both the phoneme recognition and language classification components are optimized simultaneously. Additionally, the researchers used Spleeter, a source separation tool, to isolate the vocal track from the accompaniment before processing.

Experimental Results

The system was evaluated using the DALI Dataset, which contains over 5,000 songs with word-level lyric alignments. The researchers tested the model on a "Closed-set" (English, French, German, Italian, Spanish) and an "Open-set" that included an "Others" category (Dutch, Finnish, Portuguese, Polish).

Performance comparison on the 5-language closed-set task, showing the Joint model's superior accuracy.

In the 5-language closed-set task, the Joint model achieved a record-breaking 91.7% accuracy, significantly outperforming the i-vector-based state-of-the-art (77.3%) and metadata-based identification (76.5%).

Key Insights

  • Phonemes as a Bottleneck: A "Naive End-to-End" (E2E) model that attempted to guess the language directly from audio performed poorly (59.9% accuracy). This demonstrates that forcing the model to learn phonemes as an intermediate step provides a crucial "bottleneck" that aids generalization.
  • The Power of Joint Training: Training the acoustic and language models together improved accuracy from 88.6% (2-step training) to 91.7%.
  • The "Others" Trap: While the LSTM classifier excelled in closed-set scenarios, it was prone to overfitting in open-set tasks. A simpler "Statistics" model using an SVM on phoneme distributions generalized better to unseen languages than the complex recurrent models.

Limitations and Future Work

Despite its high performance, the system faces challenges with "out-of-set" languages, often misclassifying them into known categories. Future work may explore hierarchical language modeling to handle low-resource or unseen languages more effectively. Additionally, expanding the system to include tonal languages or optimizing it for shorter audio segments (e.g., 3 seconds) remains an area for further development.

Summary updated

Abstract

Extensive works have tackled Language Identification (LID) in the speech domain, however their application to the singing voice trails and performances on Singing Language Identification (SLID) can be improved leveraging recent progresses made in other singing related tasks. This work presents a modernized phonotactic system for SLID on polyphonic music: phoneme recognition is performed with a Connectionist Temporal Classification (CTC)-based acoustic model trained with multilingual data, before language classification with a recurrent model based on the phonemes estimation. The full pipeline is trained and evaluated with a large and publicly available dataset, with unprecedented performances. First results of SLID with out-of-set languages are also presented.
BibTeX
@inproceedings{renault2021singing,
  author = {Lenny Renault and Andrea Vaglio and Romain Hennequin},
  title = {{Singing Language Identification using a Deep Phonotactic Approach}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2021},
  doi = {10.1109/icassp39728.2021.9414203},
}
Figure