Abstract
Multi-branch convolutional neural network architecture has raised lots of attention in speaker verification since the aggregation of multiple parallel branches can significantly improve performance. However, this design is not efficient enough during the inference time due to the increase of model parameters and extra operations. In this paper, we present a new multi-branch network architecture RepSPKNet that uses a re-parameterization technique. With this technique, our backbone model contains an efficient VGG-like inference state while its training state is a complicated multi-branch structure. We first introduce the specific structure of RepVGG into speaker verification and propose several variants of this structure. The performance is evaluated on VoxCeleb-based test sets. We demonstrate that both the branch diversity and the branch capacity play important roles in RepSPKNet designing. Our RepSPKNet achieves state-of-the-art performance with a 1.5982% EER and a 0.1374 minDCF on VoxCeleb1-H.
Introduction
In speaker verification, the objective is to determine whether a given voice sample belongs to a specific individual. Modern deep learning models frequently employ multi-branch architectures, such as ResNet or ResNeXt, because parallel paths enable the model to capture a wider variety of acoustic features.
However, these architectures face a significant challenge: Inference Speed. In production environments, parallel branches are computationally expensive. They increase memory bandwidth requirements and necessitate multiple kernel launches, resulting in higher latency compared to simple "plain" networks. This research addresses the trade-off by asking: Can we leverage the performance benefits of complex multi-branch networks during training while maintaining the speed of a simple linear network during deployment?
Methodology
Structural Re-parameterization
The authors utilize structural re-parameterization, a technique popularized by RepVGG, to bridge the gap between training complexity and inference efficiency. During the Training Phase, the model is constructed using "RepSPK blocks" that feature multiple parallel branches (e.g., 3x3 convolutions, 1x1 convolutions, and identity mappings). During the Inference Phase, these parallel branches are mathematically fused into a single convolution using linear algebra. For instance, a 1x1 convolution can be zero-padded to match a 3x3 kernel, allowing their weights to be summed into a single equivalent kernel.
RepSPKNet Architecture
The overall architecture consists of several stages of RepVGG blocks followed by statistical pooling and an embedding layer.
The backbone extracts features from audio spectrograms, which are then processed by a statistical pooling layer to capture global speaker characteristics. Finally, an Additive Margin Softmax (AM-Softmax) loss function is used to learn discriminative 512-dimensional speaker embeddings.
Novel Blocks: RSBA and RSBB
The researchers found that standard RepVGG blocks designed for natural images were not optimal for audio spectrograms, which have distinct frequency and time axes. They proposed two specialized variants:
- RSBA (RepSPK-A): Enhances capacity by combining a 3x3 branch with a sequential 1x1 and 3x3 branch.
- RSBB (RepSPK-B): Focuses on diversity by combining a 3x3 branch with a dilated 3x3 branch. This allows the model to capture features at multiple scales.
Experiments and Results
Datasets and Augmentation
The models were trained on the VoxCeleb2 development set, which includes over 1 million utterances from nearly 6,000 speakers. To improve robustness, the team applied 3-fold speed augmentation and added noise and reverberation using the RIRs and MUSAN datasets. Evaluation was conducted on the VoxCeleb1 test sets (O, E, and H variants).
Performance Analysis
The proposed RepSPKNet-B (utilizing RSBB blocks) demonstrated superior performance over standard baselines:
- VoxCeleb1-H (Hard): Achieved an Equal Error Rate (EER) of 1.5982% and a minimum Detection Cost Function (minDCF) of 0.1374.
- Comparative Advantage: It outperformed the widely used ECAPA-TDNN (2.01% EER) and a standard ResNet34 (1.84% EER) on the same "Hard" benchmark.
- Ablation Study: Replacing standard RepVGG blocks with the custom RSBB block yielded a relative EER improvement of approximately 10.9%.
Conclusion
This work demonstrates that "branch diversity"—specifically through the use of dilated convolutions in the RSBB block—is more critical than simple parameter capacity for speaker verification tasks. By using collapsible structures, RepSPKNet provides a "VGG-like" fast inference path that is highly optimized for hardware while maintaining the accuracy of complex ensembles. Future research may explore the integration of channel attention mechanisms or alternative pooling strategies to further enhance performance.
Summary updated