Abstract
Speaker verification systems have increasingly relied on Deep Neural Network (DNN) embeddings to represent voice identity. While standard architectures like ResNet-18 typically extract embeddings from the final hidden layer, this approach may discard valuable speaker-identifying information present in intermediate layers. This paper proposes a modified ResNet-18 architecture that utilizes multi-scale shortcut connections to aggregate features from multiple stages of the network. By concatenating average-pooled outputs from the initial pooling layer and all four residual blocks, the system generates a 1024-dimensional "deep speaker embedding." Evaluated on the VoxCeleb1 dataset, the proposed model achieved a state-of-the-art Equal Error Rate (EER) of 3.03%, representing a 21.3% relative improvement over the baseline ResNet-18.
Introduction
The primary objective of speaker verification is to determine whether a given audio sample belongs to a specific enrolled individual. Modern systems have shifted from traditional i-vectors to d-vectors—compact numerical representations extracted from Deep Neural Networks. Convolutional Neural Networks (CNNs), particularly ResNet variants, are widely used for this task. However, standard implementations often treat the network as a linear pipeline where only the final layer's output is used for the embedding. This paper argues that speaker identity is encoded across various levels of abstraction, from low-level acoustic patterns to high-level semantic traits. To capture this richness, the authors introduce a shortcut-based architecture that preserves information from earlier layers.
Proposed Methodology
The core contribution of this work is the modification of the ResNet-18 backbone to include multi-stage feature extraction.
Multi-Stage Feature Extraction and Concatenation
Instead of relying solely on the final layer, the architecture extracts features from five distinct points: the initial max-pooling layer (pool1) and the outputs of the four main residual blocks (res2 through res5). These intermediate features are processed through average pooling layers (pool2 through pool6) to ensure temporal invariance and then concatenated into a single 1024-dimensional vector.
This "shortcut" approach ensures that the final embedding contains a comprehensive set of identity features, preserving both low-level acoustic details and high-level semantic speaker traits. The model is trained end-to-end to classify 1,211 speakers using a categorical cross-entropy loss function.
Model Configuration
The specific dimensions and layer configurations are detailed in the table below. The final 1024-dimensional embedding is derived from the concatenation of outputs with varying dimensionalities (64, 64, 128, 256, and 512).
Experimental Setup
The model was trained and evaluated using the VoxCeleb1 dataset, which contains "in the wild" speech extracted from YouTube interviews.
- Training Set: 1,211 speakers (148,642 utterances).
- Test Set: 40 speakers (4,874 utterances), evaluated over 37,720 trial pairs.
- Verification Backend: During testing, the softmax layer is removed. Embeddings are compared using cosine similarity to produce a verification score. Notably, the authors achieved their results without using common "tricks" like length normalization or complex PLDA backends.
Results and Discussion
The proposed architecture demonstrated significant performance gains over established baselines. The model achieved an Equal Error Rate (EER) of 3.03% on the VoxCeleb1 test set.
Performance Comparison
The shortcut-based ResNet-18 outperformed several more complex models:
- Baseline ResNet-18: 4.30% EER (a 21.3% relative improvement).
- ResNet-34: 4.40% - 4.74% EER.
- x-vector/TDNN: 3.85% EER.
These results suggest that architectural improvements in feature aggregation can be more impactful than simply increasing network depth. Ablation studies confirmed that increasing the embedding dimensionality through layer concatenation consistently reduced the error rate, highlighting the importance of information preservation across network depths.
Conclusion
This paper demonstrates that multi-scale shortcut connections effectively preserve speaker-identifying information that is otherwise lost in deep networks. By aggregating features from multiple stages, the modified ResNet-18 achieves state-of-the-art performance on VoxCeleb1. Future research could explore combining this architecture with advanced loss functions, such as Additive Margin Softmax, to further push the boundaries of end-to-end speaker verification.
Summary updated