Anchor voiceprint recognition in live streaming via RawNet-SA and gated recurrent unit

Jiacheng Yao, Jing Zhang, Jiafeng Li, Zhuo LiEURASIP 2021 · DOI

Abstract

With the sharp booming of online live streaming platforms, some anchors seek profits and accumulate popularity by mixing inappropriate content into live programs. After being blacklisted, these anchors even forged their identities to change the platform to continue live, causing great harm to the network environment. Therefore, we propose an anchor voiceprint recognition in live streaming via RawNet-SA and gated recurrent unit (GRU) for anchor identification of live platform. First, the speech of the anchor is extracted from the live streaming by using voice activation detection (VAD) and speech separation. Then, the feature sequence of anchor voiceprint is generated from the speech waveform with the self-attention network RawNet-SA. Finally, the feature sequence of anchor voiceprint is aggregated by GRU to transform into a deep voiceprint feature vector for anchor recognition. Experiments are conducted on the VoxCeleb, CN-Celeb, and MUSAN dataset, and the competitive results demonstrate that our method can effectively recognize the anchor voiceprint in video streaming.

Introduction

The live streaming industry faces a persistent "whack-a-mole" problem: offending anchors who are blacklisted for inappropriate content often circumvent bans by registering new accounts or moving to different platforms. Identifying these individuals is difficult because they may change their visual appearance or use different setups. While voice is a relatively stable biometric, traditional speaker recognition struggles in the live streaming context. Broadcasts are rarely "clean" audio; they are cluttered with background music, game sound effects, and overlapping speech. Standard handcrafted features like Mel-frequency cepstral coefficients (MFCCs) often fail to capture the nuances of a speaker's identity when buried under such heavy noise.

System Architecture

The proposed system is designed to monitor live streams in real-time, extracting audio and comparing it against a database of known offenders. The high-level workflow involves capturing live video, isolating the audio stream, and performing deep feature extraction to generate a unique voiceprint.

System architecture for anchor voiceprint recognition in live streaming.

Methodology

The authors propose a multi-stage pipeline that moves away from manual feature engineering in favor of deep learning on raw signals:

  1. Preprocessing (VAD & Separation): To isolate the anchor's voice, the system uses VadNet for Voice Activity Detection and Spleeter (a 2-stem source separation model) to strip away background music and environmental noise.
  2. Feature Extraction (RawNet-SA): Instead of using spectrograms, the model takes raw waveforms as input. It builds on the RawNet2 architecture but introduces a Self-Attention (SA) module. While standard RawNet2 uses channel attention to weigh different frequency filters, the proposed SA module captures the relationships between different time frames, allowing the model to focus on the most discriminative parts of the speech sequence.
Feature sequence segmented by channel dimension or frame dimension to facilitate self-attention.
  1. Temporal Aggregation (GRU): Most systems use simple average pooling to turn a sequence of features into a single "voiceprint vector." This paper uses a Gated Recurrent Unit (GRU), which better preserves the temporal dependencies and phonetic patterns unique to a specific speaker.
  2. Similarity Scoring: The final identity is determined by calculating the cosine similarity between the extracted feature vector and a database of blacklisted anchor voiceprints.

Key Insights and Results

By using raw audio, the network learns its own "filter banks" (via Sinc-convolution layers), avoiding information loss inherent in pre-calculating acoustic features. The authors found that standard channel attention ignores the relationship between frames; their SA module allows the network to "look across time" to find identity-defining characteristics, significantly reducing the Equal Error Rate (EER).

Performance Benchmarks

The proposed RawNet-SA demonstrated competitive performance across several benchmarks:

  • VoxCeleb-E: Achieved an EER of 2.37% (a 0.20% improvement over the baseline RawNet2).
  • VoxCeleb-H (Hard): Achieved an EER of 4.52%, outperforming several state-of-the-art models like Thin ResNet34 and SpeakerNet.
  • CN-Celeb: In the most difficult "real-world" test, the system achieved an EER of 22.24%. While higher than the clean datasets, it represented a significant improvement over traditional methods in unconstrained environments.

Efficiency was also a priority: adding the SA module only increased inference time by approximately 0.43ms per sample compared to the baseline, making it feasible for real-time monitoring of thousands of concurrent streams.

Conclusion

While the system performs well, the jump from relatively clean VoxCeleb data to "in-the-wild" CN-Celeb data results in a significant performance drop, suggesting that domain adaptation remains a major challenge. Future work will focus on using Generative Adversarial Networks (GANs) to improve the model's ability to handle out-of-distribution data and optimizing loss functions to further separate similar-sounding speakers.

Summary updated

Abstract

Abstract With the sharp booming of online live streaming platforms, some anchors seek profits and accumulate popularity by mixing inappropriate content into live programs. After being blacklisted, these anchors even forged their identities to change the platform to continue live, causing great harm to the network environment. Therefore, we propose an anchor voiceprint recognition in live streaming via RawNet-SA and gated recurrent unit (GRU) for anchor identification of live platform. First, the speech of the anchor is extracted from the live streaming by using voice activation detection (VAD) and speech separation. Then, the feature sequence of anchor voiceprint is generated from the speech waveform with the self-attention network RawNet-SA. Finally, the feature sequence of anchor voiceprint is aggregated by GRU to transform into a deep voiceprint feature vector for anchor recognition. Experiments are conducted on the VoxCeleb, CN-Celeb, and MUSAN dataset, and the competitive results demonstrate that our method can effectively recognize the anchor voiceprint in video streaming.
BibTeX
@inproceedings{yao2021anchor,
  author = {Jiacheng Yao and Jing Zhang and Jiafeng Li and Zhuo Li},
  title = {{Anchor voiceprint recognition in live streaming via RawNet-SA and gated recurrent unit}},
  booktitle = {EURASIP},
  year = {2021},
  doi = {10.1186/s13636-021-00234-3},
}
Figure