Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification

Bei Liu, Yanmin Qian

TL;DR

This paper introduces a highly memory-efficient training strategy for deep speaker verification systems, achieving up to a 16.2x reduction in GPU memory usage with virtually no loss in accuracy. By combining reversible neural network architectures with dynamic 8-bit optimizer quantization, the researchers demonstrate that state-of-the-art, ultra-deep models can be trained on consumer-grade hardware (like a single RTX 2080Ti) instead of expensive enterprise GPU clusters.

What Problem Does This Paper Address?

In speaker verification (SV)—the technology behind voice biometrics—the trend has leaned heavily toward deeper and larger neural network extractors (such as 233-layer DF-ResNets or 518-layer ResNets). While these deep architectures yield exceptional accuracy, they hit a massive "memory wall."

During standard training, deep learning frameworks cache intermediate activations during the forward pass so they can be used to calculate gradients during the backward pass. As networks grow deeper, this activation cache scales linearly, consuming the vast majority of GPU memory. Consequently, training state-of-the-art SV models typically requires massive clusters of high-end enterprise GPUs (like NVIDIA V100s or A100s), making this research inaccessible to smaller labs and independent developers.

Methods

The authors first systematically analyzed GPU memory allocation during training. As shown in the pie charts below, activations alone account for over 90% of the total memory overhead:

Fig. 1: GPU memory allocation during the training process for ResNet34 and DF-ResNet56 in speaker verification.

To tackle this, the authors proposed a two-pronged strategy targeting the two largest memory sinks: activations and optimizer states.

1. Reversible Neural Networks (RevNets & DF-RevNets)

Instead of caching activations, the authors designed networks using reversible operators. In a reversible layer, the inputs can be mathematically reconstructed from the outputs during the backward pass ($x = f^{-1}(y)$).

Fig. 2: The comparison between reversible and non-reversible operators during the back-propagation.

The authors introduced two architectural flavors:

  • Type I (Partially Reversible): Converts standard residual blocks into reversible blocks via an additive coupling mechanism but retains standard, irreversible downsampling layers.
  • Type II (Fully Reversible): Replaces standard downsampling (like strided convolutions) with an invertible tensor-reshaping operation inspired by pixel-shuffle techniques, ensuring the entire network remains memory-flat regardless of depth.

2. 8-bit Dynamic Quantization for Optimizers

Stateful optimizers like SGD with momentum or Adam store historical gradient statistics in 32-bit floating-point formats. The authors introduced a dynamic quantization scheme that compresses these states to 8-bit integers. To prevent precision loss from outliers, they implemented block-wise normalization and utilized a dynamic, tree-based 8-bit data type that adaptively adjusts fraction and exponent bits to preserve accuracy across varying magnitudes.

8-bit dynamic quantization process

Datasets

The approaches were validated using the standard VoxCeleb1 and VoxCeleb2 benchmark datasets, which consist of voice clips from over 6,000 celebrities on YouTube.

  • Training: VoxCeleb2 development set (~2,200 hours of speech).
  • Data Augmentation: The training data was tripled by applying speed perturbation (0.9x and 1.1x) alongside online noise addition using the MUSAN corpus and Room Impulse Response (RIR) reverberations.
  • Evaluation: Tested against the official Vox1-O, Vox1-E, and Vox1-H evaluation trials.

Key Insights and Ideas

  • Decoupling Depth from Memory: The most profound insight is that reversible architectures completely flatten the activation memory curve. Memory consumption remains virtually constant whether a network has 50 layers or 300 layers.
  • The Space-for-Time Trade-off: Reversible networks trade a small amount of computational time (recomputing activations on the fly) to save massive amounts of physical space. For resource-constrained environments, this trade-off is incredibly favorable.
  • Basic Blocks Punch Above Their Weight: Interestingly, the authors found that reversible networks built using simpler "Basic Blocks" achieved comparable performance to standard networks using complex "Bottleneck Blocks," but with fewer parameters.

Key Results

  • Massive Memory Compression: The proposed framework achieved up to 16.2x memory savings compared to vanilla baselines.
  • Batch Size Expansion: For a heavy model like ResNet152, the maximum allowable batch size on an 11GB GPU skyrocketed from 22 to 295 samples.
  • Identical Accuracy: Across all evaluation sets (Vox1-O, E, H), the Equal Error Rate (EER) of the memory-efficient variants remained on par with the original 32-bit baseline models.
  • Democratic Hardware Requirements: Instead of requiring 4x A100 GPUs to train a massive DF-ResNet233 model, the reversible equivalent (DF-RevNet377) was successfully trained using just two consumer-level RTX 2080Ti GPUs.

Limitations and Future Work

The primary limitation of this approach is increased training time. Because activations must be recomputed during the backward pass, there is an added computational overhead. For example, training the baseline DF-ResNet233 on 4x A100 GPUs took roughly 2.5 days, whereas training the memory-efficient variant on 2x 2080Ti GPUs took about 6 days (though this is also heavily influenced by the raw hardware performance gap).

Future work will likely focus on optimizing the computational efficiency of the reversible operations to bridge this training time gap, as well as extending this framework to self-supervised speech models and Transformers.

Summary updated

Abstract

Recent speaker verification (SV) systems have shown a trend toward adopting deeper speaker embedding extractors. Although deeper and larger neural networks can significantly improve performance, their substantial memory requirements hinder training on consumer GPUs. In this paper, we explore a memory-efficient training strategy for deep speaker embedding learning in resource-constrained scenarios. Firstly, we conduct a systematic analysis of GPU memory allocation during SV system training. Empirical observations show that activations and optimizer states are the main sources of memory consumption. For activations, we design two types of reversible neural networks which eliminate the need to store intermediate activations during back-propagation, thereby significantly reducing memory usage without performance loss. For optimizer states, we introduce a dynamic quantization approach that replaces the original 32-bit floating-point values with a dynamic tree-based 8-bit data type. Experimental results on VoxCeleb demonstrate that the reversible variants of ResNets and DF-ResNets can perform training without the need to cache activations in GPU memory. In addition, the 8-bit versions of SGD and Adam save 75% of memory costs while maintaining performance compared to their 32-bit counterparts. Finally, a detailed comparison of memory usage and performance indicates that our proposed models achieve up to 16.2x memory savings, with nearly identical parameters and performance compared to the vanilla systems. In contrast to the previous need for multiple high-end GPUs such as the A100, we can effectively train deep speaker embedding extractors with just one or two consumer-level 2080Ti GPUs.
BibTeX
@misc{liu2024memory,
  author = {Bei Liu and Yanmin Qian},
  title = {{Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification}},
  year = {2024},
  eprint = {2412.01195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01195},
}
Figure