A Health-friendly Speaker Verification System Supporting Mask Wearing

Chaotao Chen, Di Jiang, Jinhua Peng, Rongzhong Lian, Chen Jason Zhang, Qian Xu, Lixin Fan, Qiang Yang

Abstract

The COVID-19 pandemic highlighted the limitations of facial recognition systems when users wear face masks. To address this, researchers from WeBank and HKUST developed a speaker verification (SV) system that remains accurate even when speech is muffled by masks. By combining a ResNet-34 feature extractor with a Multi-Head Attention (MHA) mechanism and Additive Margin Softmax (AM-Softmax) loss, the system achieves a remarkably low Equal Error Rate (EER) of 0.076%. This solution provides a secure, touchless, and health-friendly authentication method suitable for mobile deployment.

Introduction

Traditional biometric authentication, particularly facial recognition, often requires users to remove masks, creating potential health risks in public environments. Speaker verification (SV) offers a natural alternative, but physical barriers like masks can distort acoustic patterns. This research focuses on building a robust SV system capable of handling these "complex scenarios" without compromising security or user safety.

Methodology

The system architecture follows a sophisticated three-stage pipeline designed to extract robust speaker identities from potentially distorted audio.

  1. Speech Processing: Raw audio undergoes Voice Activity Detection (VAD) to remove silence. The remaining signal is converted into log-scaled filter-banks and normalized using Cepstral Mean and Variance Normalization (CMVN).
  2. Local Acoustic Pattern Extractor: A modified ResNet-34 convolutional neural network extracts frame-level representations, capturing the unique textures of a speaker's voice in short time slices.
  3. Multi-Head Attention (MHA) Embedding: Instead of simple statistical pooling, the system uses MHA to aggregate frame-level features into a single utterance-level embedding. This allows the model to focus on the most discriminative parts of the speech, effectively ignoring frequencies muffled by a mask.
System architecture of the proposed speaker verification pipeline.

To enhance reliability, the model is optimized with Additive Margin Softmax (AM-Softmax) loss, which forces a wider mathematical margin between different speakers in the embedding space.

Datasets and Training

The model was trained on a diverse collection of over 40,000 speakers to ensure broad generalization across languages and accents. The training set included:

  • VoxCeleb1 & VoxCeleb2: Large-scale celebrity speech datasets.
  • Aishell-1 & Aishell-2: Massive open-source Mandarin speech corpora.
  • MAGIDATA: Additional robust speech data.

Key Results and Usability

The system demonstrates high performance and practical efficiency for real-world use:

  • Accuracy: Achieved an EER of 0.076% in complex testing scenarios.
  • Speed: Requires only 10 seconds of speech for registration and 3 seconds for verification.
  • Mobile Deployment: The system is optimized for mobile use, as demonstrated by its implementation in a WeChat mini-program for secure login and payments.
User interface for registration and verification in the mobile application.

Conclusion and Future Work

This "health-friendly" speaker verification system successfully balances security and hygiene. By utilizing attention-based pooling and margin-based learning, it overcomes the acoustic challenges posed by face masks. Future research will focus on further optimizing the model for low-power mobile devices and improving performance in high-noise environments like airports or hospitals.

Summary updated

Abstract

We demonstrate a health-friendly speaker verification system for voice-based identity verification on mobile devices. The system is built upon a speech processing module, a ResNet-based local acoustic feature extractor and a multi-head attention-based embedding layer, and is optimized under an additive margin softmax loss for discriminative speaker verification. It is shown that the system achieves superior performance no matter whether there is mask wearing or not. This characteristic is important for speaker verification services operating in regions affected by the raging coronavirus pneumonia. With this demonstration, the audience will have an in-depth experience of how the accuracy of bio-metric verification and the personal health are simultaneously ensured. We wish that this demonstration would boost the development of next-generation bio-metric verification technologies.
BibTeX
@inproceedings{chen2021health,
  author = {Chaotao Chen and Di Jiang and Jinhua Peng and Rongzhong Lian and Chen Jason Zhang and Qian Xu and Lixin Fan and Qiang Yang},
  title = {{A Health-friendly Speaker Verification System Supporting Mask Wearing}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2021},
  doi = {10.1609/aaai.v35i18.17994},
}
Figure