Abstract
As the Metaverse expands, identifying the human identity behind virtual avatars becomes a critical security and personalization challenge. This paper presents a robust Singer Identification (SID) system designed to overcome the interference of background accompaniment and the "producer effect." By integrating frame-level Mel-spectrograms, noise-resistant timbral X-vectors, and middle-level perceptual features through a Convolutional Recurrent Neural Network (CRNN), the proposed model achieves a state-of-the-art F1 score of 0.86 on the Artist20 benchmark.
Introduction
In the emerging Metaverse, users interact through digital avatars, often participating in virtual concerts, karaoke, or parody performances. A significant technical hurdle in these environments is Singer Identification (SID)—the ability to verify the actual human singer behind a digital mask. Unlike traditional speaker recognition, SID must contend with complex musical backgrounds and the physiological similarities between trained vocalists. This research proposes a multi-feature fusion approach to create a more resilient "vocal fingerprint" for virtual environments.
The Challenge of Singer Identification
SID is inherently more difficult than standard speech recognition due to several factors:
- Accompaniment Noise: Most SID models struggle to separate the singer's voice from the background instrumentation. Models often inadvertently learn the "style" of the production or specific instruments rather than the singer's unique vocal characteristics.
- The Producer Effect: When models are trained on specific albums, they often memorize the specific mixing and mastering settings of that album. This leads to high accuracy on known tracks but poor generalization to new recordings by the same artist.
- Vocal Similarity: Human singing voices often share similar ranges and timbres, requiring high-dimensional features to distinguish between different performers.
Proposed Methodology
The authors introduce a multi-stream architecture based on a Convolutional Recurrent Neural Network (CRNN) that fuses three distinct categories of audio data:
1. Frame-Level Features
The model utilizes Mel-spectrograms to provide a standard time-frequency representation of the audio signal, capturing the fundamental melodic and harmonic structure.
2. Timbral Features (X-vectors)
Borrowed from speaker verification research, X-vectors are deep neural network embeddings designed to be robust against environmental noise and reverberation. In this context, they serve as a noise-resistant representation of the singer's vocal tract characteristics, helping the model ignore the backing track.
3. Middle-Level Perceptual Features
These features represent higher-level musical qualities such as melodiousness, rhythmic stability, and tonal stability. The researchers employed transfer learning from an Inception V3 model (originally trained for music emotion recognition). They tested features extracted from different layers (L3, L4, and L5) to determine which provided the most relevant information for identification.
Architecture Design
The system employs a CNN to fuse the Mel-spectrograms and X-vectors. This fused output is then processed by Gated Recurrent Units (GRU), which are specifically suited for capturing the temporal and sequential nature of singing. Finally, a dense layer performs the classification across the artist pool.
Experimental Evaluation
The model was evaluated using the Artist20 dataset, which consists of 1,413 tracks from 20 artists across 6 albums each. To ensure the model's robustness against the "producer effect," the authors utilized an album-split evaluation strategy. By training on certain albums and testing on entirely different ones, the researchers ensured the model was identifying the singer rather than the album's production style.
Key Results and Analysis
The proposed CRNN+X-vector+L4 model significantly outperformed existing benchmarks:
- Performance: The model achieved a peak F1 score of 0.86 and an average of 0.81, a substantial improvement over the previous state-of-the-art (CRNNM), which scored 0.73.
- The Impact of X-vectors: Integrating X-vectors improved the F1 score by 0.11, confirming that speaker-recognition embeddings are highly effective at filtering out musical "noise."
- Optimal Feature Extraction: The study found that Layer 4 (L4) of the Inception V3 model provided the ideal balance of information. Layer 3 retained too much noise (under-fitting), while Layer 5 lost identifying details through excessive convolution (over-fitting).
- Multicollinearity: Interestingly, combining all perceptual layers (L3+L4+L5) decreased performance. This suggests that redundant features can confuse the network, a phenomenon known as multicollinearity.
Conclusion and Future Work
This research demonstrates that fusing timbral embeddings with middle-level perceptual features creates a highly effective system for identifying singers in noisy, music-heavy environments like the Metaverse. While the results on the Artist20 dataset are promising, future work will focus on scaling the system to handle thousands of potential singers and optimizing the model for real-time processing during live virtual events. Additionally, the authors suggest this multi-feature fusion approach could enhance other Music Information Retrieval (MIR) tasks, such as genre classification and similarity-based search engines.
Summary updated