Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification

Bei Liu, Yanmin Qian

Introduction

In the field of Speaker Verification (SV), model depth has become a primary driver of performance. State-of-the-art systems often utilize architectures with hundreds of layers, such as ResNet-518, to achieve high accuracy in challenges like VoxSRC. However, scaling these models introduces a significant "memory wall." Standard training procedures require storing intermediate activations for every layer to compute gradients during back-propagation, leading to massive VRAM requirements that often exceed the capacity of consumer-grade hardware. This paper presents a framework to democratize deep SV training by drastically reducing GPU memory consumption without compromising verification accuracy.

The GPU Memory Bottleneck

The primary obstacle to training deeper models is not the number of parameters, but the storage of activations. Analysis of deep SV models reveals that activations can account for over 90% of total GPU memory usage during training. This necessitates the use of high-end enterprise GPUs like the NVIDIA A100 or forces researchers to use sub-optimally small batch sizes.

Breakdown of GPU memory allocation in deep speaker verification models, showing that activations dominate usage.

Proposed Methodology

To address this bottleneck, the authors target the two largest memory consumers: activations and optimizer states.

Reversible Neural Networks (RevNets)

The core innovation involves replacing standard residual blocks with reversible blocks. In a standard network, the input $x$ must be stored to compute the gradient of the output $y$ with respect to the weights $w$. In a reversible network, the layers are designed such that the input $x$ can be mathematically reconstructed from the output $y$ during the backward pass ($x = f^{-1}(y)$).

Comparison between standard back-propagation, which stores activations, and reversible back-propagation, which recomputes them.

By recomputing activations on the fly, the memory cost for activations becomes nearly independent of the network's depth. The authors implemented two types of reversible architectures: Type I, which maintains standard downsampling, and Type II (DF-RevNet), which utilizes a fully reversible "pixel compression" reshaping technique for downsampling.

8-bit Dynamic Optimizer Quantization

While activations are the largest hog, optimizer states (like momentum and variance in Adam) also consume significant memory when stored in 32-bit floats. The framework introduces a dynamic tree-based 8-bit data type for these states. This format uses an indicator bit to adaptively adjust precision for different value ranges. To maintain accuracy and handle outliers, the authors employ tensor chunking, breaking weights into blocks of 2048 for localized quantization.

Experimental Results

The proposed methods were evaluated on the VoxCeleb benchmarks using standard augmentation techniques (RIR, MUSAN, and speed perturbation).

Memory Savings and Hardware Accessibility

The combined approach achieved up to a 16.2x reduction in memory requirements. This efficiency allows a 233-layer DF-ResNet—which normally requires a cluster of A100 GPUs—to be trained on just two consumer-level RTX 2080Ti GPUs. For a ResNet-152 model, the maximum batch size on an 11GB GPU increased from 22 to 295 samples.

Performance Parity

Crucially, these memory savings do not come at the cost of accuracy. The Equal Error Rate (EER) remained virtually identical to vanilla baselines across Vox1-O, Vox1-E, and Vox1-H test sets. For instance, the ResNet-152 variants maintained an EER of approximately 0.55% on Vox1-O.

Conclusion and Future Work

This memory-efficient framework successfully bridges the gap between massive model architectures and accessible hardware. While the reversible approach increases training time by 30-50% due to re-computation, the trade-off enables the exploration of significantly deeper models on modest infrastructure. Future research will look toward applying these techniques to large-scale speech foundation models and combining them with model pruning and distillation for further optimization.

Summary updated

Abstract

Recent speaker verification (SV) systems have shown a trend toward adopting deeper speaker embedding extractors. Although deeper and larger neural networks can significantly improve performance, their substantial memory requirements hinder training on consumer GPUs. A number of memory-efficient training techniques have been explored in the MLSys community, including reversible networks and optimizer state quantization, which aim to reduce activation and optimizer memory usage. However, these general techniques have rarely been adapted or systematically analyzed for SV models operating on time–frequency representations. In this paper, we revisit these two classic directions and introduce a joint memory-efficient training strategy tailored for deep speaker embedding learning under resource-constrained conditions. Specifically, we (1) design two types of reversible architectures for SV, including a reshape-based reversible downsampling mechanism (Type II RevNets) to enhance the degree of reversibility across all layers, and (2) propose a dynamic, tree-based 8-bit quantization scheme for optimizer states that preserves numerical stability under heavy-tailed parameter distributions. The proposed joint reversible–quantized optimizer framework achieves up to 16.2x memory savings while maintaining nearly identical parameters and performance compared to the original systems. Extensive experiments on VoxCeleb show that our method enables training of deep ResNet and DF-ResNet models using only one or two consumer-grade GPUs, without sacrificing performance.
BibTeX
@misc{liu2024memory,
  author = {Bei Liu and Yanmin Qian},
  title = {{Memory-Efficient Training for Deep Speaker Embedding Learning in Speaker Verification}},
  year = {2024},
  eprint = {2412.01195},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2412.01195},
}
Figure