Shortcut Connections Based Deep Speaker Embeddings for End-to-End Speaker Verification System

Soonshin Seo, Daniel Jun Rim, Minkyu Lim, Donghyun Lee, Hosung Park, Junseok Oh, Changmin Kim, Ji-Hwan KimInterspeech 2019 · DOI

Abstract

Speaker verification systems have increasingly relied on Deep Neural Network (DNN) embeddings to represent voice identity. While standard architectures like ResNet-18 typically extract embeddings from the final hidden layer, this approach may discard valuable speaker-identifying information present in intermediate layers. This paper proposes a modified ResNet-18 architecture that utilizes multi-scale shortcut connections to aggregate features from multiple stages of the network. By concatenating average-pooled outputs from the initial pooling layer and all four residual blocks, the system generates a 1024-dimensional "deep speaker embedding." Evaluated on the VoxCeleb1 dataset, the proposed model achieved a state-of-the-art Equal Error Rate (EER) of 3.03%, representing a 21.3% relative improvement over the baseline ResNet-18.

Introduction

The primary objective of speaker verification is to determine whether a given audio sample belongs to a specific enrolled individual. Modern systems have shifted from traditional i-vectors to d-vectors—compact numerical representations extracted from Deep Neural Networks. Convolutional Neural Networks (CNNs), particularly ResNet variants, are widely used for this task. However, standard implementations often treat the network as a linear pipeline where only the final layer's output is used for the embedding. This paper argues that speaker identity is encoded across various levels of abstraction, from low-level acoustic patterns to high-level semantic traits. To capture this richness, the authors introduce a shortcut-based architecture that preserves information from earlier layers.

Proposed Methodology

The core contribution of this work is the modification of the ResNet-18 backbone to include multi-stage feature extraction.

Multi-Stage Feature Extraction and Concatenation

Instead of relying solely on the final layer, the architecture extracts features from five distinct points: the initial max-pooling layer (pool1) and the outputs of the four main residual blocks (res2 through res5). These intermediate features are processed through average pooling layers (pool2 through pool6) to ensure temporal invariance and then concatenated into a single 1024-dimensional vector.

Proposed Deep Speaker Embedding architecture with multi-stage shortcut connections.

This "shortcut" approach ensures that the final embedding contains a comprehensive set of identity features, preserving both low-level acoustic details and high-level semantic speaker traits. The model is trained end-to-end to classify 1,211 speakers using a categorical cross-entropy loss function.

Model Configuration

The specific dimensions and layer configurations are detailed in the table below. The final 1024-dimensional embedding is derived from the concatenation of outputs with varying dimensionalities (64, 64, 128, 256, and 512).

Detailed configuration and output dimensions of the modified ResNet-18 model.

Experimental Setup

The model was trained and evaluated using the VoxCeleb1 dataset, which contains "in the wild" speech extracted from YouTube interviews.

  • Training Set: 1,211 speakers (148,642 utterances).
  • Test Set: 40 speakers (4,874 utterances), evaluated over 37,720 trial pairs.
  • Verification Backend: During testing, the softmax layer is removed. Embeddings are compared using cosine similarity to produce a verification score. Notably, the authors achieved their results without using common "tricks" like length normalization or complex PLDA backends.

Results and Discussion

The proposed architecture demonstrated significant performance gains over established baselines. The model achieved an Equal Error Rate (EER) of 3.03% on the VoxCeleb1 test set.

Performance Comparison

The shortcut-based ResNet-18 outperformed several more complex models:

  • Baseline ResNet-18: 4.30% EER (a 21.3% relative improvement).
  • ResNet-34: 4.40% - 4.74% EER.
  • x-vector/TDNN: 3.85% EER.

These results suggest that architectural improvements in feature aggregation can be more impactful than simply increasing network depth. Ablation studies confirmed that increasing the embedding dimensionality through layer concatenation consistently reduced the error rate, highlighting the importance of information preservation across network depths.

Conclusion

This paper demonstrates that multi-scale shortcut connections effectively preserve speaker-identifying information that is otherwise lost in deep networks. By aggregating features from multiple stages, the modified ResNet-18 achieves state-of-the-art performance on VoxCeleb1. Future research could explore combining this architecture with advanced loss functions, such as Additive Margin Softmax, to further push the boundaries of end-to-end speaker verification.

Summary updated

BibTeX
@inproceedings{seo2019shortcut,
  author = {Soonshin Seo and Daniel Jun Rim and Minkyu Lim and Donghyun Lee and Hosung Park and Junseok Oh and Changmin Kim and Ji-Hwan Kim},
  title = {{Shortcut Connections Based Deep Speaker Embeddings for End-to-End Speaker Verification System}},
  booktitle = {Interspeech},
  year = {2019},
  doi = {10.21437/interspeech.2019-2195},
}
Figure