MetaSID: Singer Identification with Domain Adaptation for Metaverse

Xulong Zhang, Jianzong Wang, Ning Cheng, Jing Xiao

Abstract

Singer Identification (SID) is a critical component for personalized experiences in the Metaverse, yet it faces significant challenges due to the "album effect"—a performance drop caused by domain shifts between studio recordings and live environments. This paper introduces MetaSID, a framework that leverages a Convolutional Recurrent Neural Network (CRNN) combined with Domain Adaptation (DA) techniques to achieve robust singer identification. By evaluating three DA strategies—Maximum Mean Discrepancy (MMD), Gradient Reversal (RevGrad), and the Contrastive Adaptation Network (CAN)—we demonstrate that class-aware adaptation significantly improves cross-domain accuracy. Our proposed CRNN-CAN model achieves a state-of-the-art F1 score of 0.83 on the Artist20 benchmark, effectively bridging the gap between studio and live audio distributions.

Introduction

In the emerging Metaverse, identifying singers across diverse virtual venues and recording conditions is essential for content organization and user interaction. However, traditional SID models often suffer from the "album effect." This phenomenon occurs because songs within a single album share production characteristics—such as specific microphones, mixing styles, and acoustic environments—that differ from those of other albums or live performances. When a model trained on studio data is deployed in a different "domain" (like a virtual concert), the distribution shift leads to a sharp decline in recognition accuracy. MetaSID addresses this by learning domain-invariant features that capture the intrinsic characteristics of a singer's voice rather than the extrinsic noise of the production environment.

Methodology

Backbone Architecture

The core of MetaSID is a Convolutional Recurrent Neural Network (CRNN). This architecture is specifically chosen for its ability to process audio signals:

  • CNN Blocks: Four convolutional blocks extract local spectro-temporal features, effectively capturing the "timbre" or unique quality of the singer's voice.
  • GRU Blocks: Two Gated Recurrent Units (GRU) model the temporal dependencies, capturing the "prosody"—the rhythm and melodic patterns that define a singer's style over time.

Domain Adaptation Strategies

To overcome the album effect, the researchers integrated three distinct DA strategies into the CRNN framework:

  1. CRNN-MMD (Maximum Mean Discrepancy): This approach minimizes the statistical distance between the source (studio) and target (live/new album) feature distributions. It adds a domain loss to the network to ensure the global distributions are as similar as possible.
  2. CRNN-RevGrad (Gradient Reversal): An adversarial method employing a domain discriminator. The discriminator attempts to identify the source domain of a feature, while a gradient reversal layer forces the feature extractor to produce features that deceive the discriminator, resulting in domain-invariant representations.
  3. CRNN-CAN (Contrastive Adaptation Network): The most advanced strategy, CAN is "class-aware." While MMD and RevGrad align global distributions, CAN ensures that features of the same singer are clustered together across domains (minimizing intra-class discrepancy) while features of different singers are pushed apart (maximizing inter-class discrepancy).

Experimental Setup

Dataset

The study utilized the Artist20 dataset, a standard benchmark in Music Information Retrieval (MIR), consisting of 1,413 songs from 20 artists across 6 albums each.

Evaluation Protocol

To strictly test for the album effect, the authors employed an album split methodology. The models were trained on 4 albums per artist and tested on the remaining 2. This setup prevents the model from relying on album-specific production quirks, forcing it to generalize across different recording environments.

Results and Discussion

The experimental results demonstrate a clear hierarchy in the effectiveness of domain adaptation techniques. The baseline CRNN, without any adaptation, achieved an F1 score of only 0.60, highlighting the severity of the album effect.

Model F1 (Avg) Improvement over Baseline
Baseline CRNN 0.60 -
CRNN-MMD 0.74 +0.14
CRNN-RevGrad 0.81 +0.21
CRNN-CAN 0.83 +0.23

The Importance of Class-Awareness

The superior performance of CRNN-CAN (0.83 F1) underscores the importance of class-aware adaptation. Global methods like MMD or RevGrad risk "misalignment," where the features of one singer in the source domain might be mapped to a different singer in the target domain. By explicitly considering the singer's identity during the adaptation process, CAN ensures that the learned features are both domain-invariant and discriminative.

Conclusion

MetaSID provides a robust framework for singer identification in the face of significant domain shifts. By combining the temporal modeling of CRNNs with class-aware contrastive adaptation, the model successfully mitigates the album effect. While the current study focuses on album splits, future work will explore even more complex Metaverse environments, including 3D spatial audio and varying virtual acoustics, to further enhance the system's real-world applicability.

Summary updated

BibTeX
@inproceedings{zhang2022metasid,
  author = {Xulong Zhang and Jianzong Wang and Ning Cheng and Jing Xiao},
  title = {{MetaSID: Singer Identification with Domain Adaptation for Metaverse}},
  booktitle = {IJCNN},
  year = {2022},
  url = {https://openreview.net/forum?id=X5LI1hWa9At},
}
Figure