On Batching Variable Size Inputs for Training End-to-End Speech Enhancement Systems

Philippe Gonzalez, Tommy Sonne Alstrøm, Tobias May

Introduction

In the development of end-to-end speech enhancement systems, researchers often face a fundamental conflict between the variable nature of audio signals and the rigid requirements of modern deep learning hardware. Standard training pipelines utilize mini-batch stochastic gradient descent, which requires stacking multiple samples into a single tensor. Because speech utterances naturally vary in duration, the common solution is to apply zero-padding—adding silence to shorter signals to match the longest sequence in the batch.

However, this approach introduces two primary inefficiencies. First, computational resources are wasted as the GPU performs calculations on "dead air" (zeros), increasing training time and memory consumption. Second, there is a long-standing belief in the machine learning community that data must be shuffled randomly (independent and identically distributed, or i.i.d. sampling) to ensure optimal generalization. Maximizing randomness, however, also maximizes the variance in sequence lengths within a batch, thereby maximizing the necessary padding. This paper investigates whether alternative batching strategies can improve efficiency without compromising the model's ability to generalize to unseen acoustic environments.

Methodology

The study evaluates three distinct batching strategies designed to manage variable-length inputs:

  1. Random Batching: The baseline approach where sequences are sampled randomly. This maximizes data shuffling but results in high Zero-Padding Rates (ZPR).
  2. Sorted Batching: Sequences are sorted by length before being partitioned into batches. This minimizes padding to nearly zero but significantly reduces the randomness of samples within a single batch.
  3. Bucket Batching: A hybrid approach where sequences are grouped into "buckets" based on their duration. Batches are then drawn from within these buckets, providing a middle ground between efficiency and randomization.

These strategies were tested using two batch-sizing methods: Fixed Batching (a constant number of sequences per batch) and Dynamic Batching (a constant total duration of audio per batch). To ensure the model—Conv-TasNet—did not learn from the padded regions, the researchers implemented a masked loss function, effectively nullifying the contribution of zero-padded segments during backpropagation.

Experimental Design

The models were trained on a 10-hour dataset synthesized from LibriSpeech (clean speech), TAU urban acoustic scenes (noise), and University of Surrey room impulse responses (reverberation).

To rigorously test generalization, the authors utilized two evaluation sets:

  • Matched Condition: New mixtures created using the same underlying databases as the training set.
  • Mismatched Condition: Mixtures created from entirely different corpora, including TIMIT, WSJ, and Clarity for speech, and NOISEX and DEMAND for noise. This condition is critical for determining if batching strategies affect the model's robustness to unseen speakers and environments.

Key Results and Analysis

Efficiency and Throughput

The results demonstrate that dynamic batching combined with sorted or bucketed strategies offers significant computational advantages. Sorted batching achieved a Zero-Padding Rate (ZPR) as low as 0.4%, compared to 24.1% for random batching at similar scales. This efficiency translated into a 18% to 26% reduction in total training time. Furthermore, dynamic batching prevented memory spikes, as it ensures the total number of samples (and thus the memory footprint) remains relatively constant regardless of individual sequence lengths.

The Impact of Batch Size on Generalization

A consistent trend emerged across all experiments: smaller batch sizes led to superior performance. This was particularly evident in the mismatched evaluation conditions, where models trained with smaller dynamic batches (e.g., 2 seconds of audio) showed the highest improvements in Signal-to-Noise Ratio ($\Delta$SNR) and Perceptual Evaluation of Speech Quality ($\Delta$PESQ). This supports the theory that smaller batches provide a beneficial regularization effect, helping the model generalize better to out-of-distribution data.

Challenging the i.i.d. Assumption

Perhaps the most surprising finding was that Sorted Batching—despite its lack of intra-batch randomization—performed as well as Random Batching. This suggests that the requirement for i.i.d. sampling is less stringent than previously thought for speech enhancement tasks, provided that the batches themselves are shuffled at the start of each epoch. The global randomization of the training process appears sufficient to maintain performance.

Conclusion

This research provides a practical roadmap for optimizing the training of speech enhancement models. By adopting dynamic, sorted batching, researchers can significantly reduce training time and GPU memory usage without any measurable loss in model quality. The study highlights that the trade-off between efficiency and generalization is largely manageable; with the right batching strategy and a masked loss function, one can achieve both high throughput and robust performance. While these findings were established on a 10-hour dataset using Conv-TasNet, they offer a compelling baseline for scaling up to larger foundation models and other variable-length sequence tasks.

Summary updated

Abstract

The performance of neural network-based speech enhancement systems is primarily influenced by the model architecture, whereas training times and computational resource utilization are primarily affected by training parameters such as the batch size. Since noisy and reverberant speech mixtures can have different duration, a batching strategy is required to handle variable size inputs during training, in particular for state-of-the-art end-to-end systems. Such strategies usually strive for a compromise between zero-padding and data randomization, and can be combined with a dynamic batch size for a more consistent amount of data in each batch. However, the effect of these strategies on resource utilization and more importantly network performance is not well documented. This paper systematically investigates the effect of different batching strategies and batch sizes on the training statistics and speech enhancement performance of a Conv-TasNet, evaluated in both matched and mismatched conditions. We find that using a small batch size during training improves performance in both conditions for all batching strategies. Moreover, using sorted or bucket batching with a dynamic batch size allows for reduced training time and GPU memory usage while achieving similar performance compared to random batching with a fixed batch size.
BibTeX
@inproceedings{gonzalez2023batching,
  author = {Philippe Gonzalez and Tommy Sonne Alstrøm and Tobias May},
  title = {{On Batching Variable Size Inputs for Training End-to-End Speech Enhancement Systems}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2023},
  url = {https://openreview.net/forum?id=ts9yR55lKI},
}
Figure