In standard deep learning, small batch sizes are a well-established heuristic. As noted by Ossi Räisä et al. in Subsampling is not Magic: Why Large Batch Sizes Work for Differentially Private Stochastic Optimisation, smaller batches introduce stochasticity that helps models escape local minima and generalize better to unseen data.
However, when training with Differential Privacy (DP), this heuristic completely breaks down. Under DP constraints, small batch sizes often lead to catastrophic utility loss, forcing a paradigm shift toward massive "mega-batches."
The Root Cause: Constant Noise vs. Growing Signal
To understand why private training requires a fundamentally different approach to batch sizing, we must look at how privacy is enforced. In Differentially Private Stochastic Gradient Descent (DP-SGD), introduced by Martı́n Abadi et al. in Deep Learning with Differential Privacy, individual gradients are clipped to a maximum threshold to bound their sensitivity, and random Gaussian noise is injected to obscure any single individual's contribution.
The critical mathematical insight, highlighted by Rohan Anil et al. in Large-Scale Differentially Private BERT and Natalia Ponomareva et al. in How to DP-fy ML: A Practical Guide to Machine Learning with Differential Privacy, is that the magnitude of the injected privacy noise is independent of the batch size. It depends strictly on the sensitivity bound and the target privacy budget ($\epsilon, \delta$).
When gradients are computed over a small batch, the true underlying gradient signal is weak, and it easily becomes completely overwhelmed by the fixed privacy noise. By scaling the batch size to extreme levels, the true gradient signals from hundreds of thousands or millions of examples are aggregated together. As Carson Denison et al. explain in Private Ad Modeling with DP-SGD, averaging the step over a larger number of individual gradients effectively dilutes the relative impact of the fixed Gaussian noise. This drastically improves the Signal-to-Noise Ratio (SNR) of the optimization step without consuming any additional privacy budget.
Small Batch: [ Weak Gradient Signal ] + [ FIXED PRIVACY NOISE ] --> Low SNR (Noise dominates)
Mega-Batch: [ MASSIVE GRADIENT SIGNAL ] + [ FIXED PRIVACY NOISE ] --> High SNR (Signal dominates)
From Theory to Mega-Batches
Early work by Martı́n Abadi et al. suggested that setting the batch size proportional to the square root of the total dataset size could balance privacy and utility. However, as models and datasets have scaled to foundation-model proportions, practitioners have pushed batch sizes far beyond this threshold to achieve acceptable utility.
For instance, Rohan Anil et al. successfully trained BERT-Large using "mega-batches" of 2 million examples. Similarly, in the domain of computer vision, Tom Sander et al. demonstrated in Differentially Private Representation Learning via Image Captioning that scaling batch sizes to over one million samples allows for stable loss trajectories and high-utility representation learning from scratch. In generative modeling, Alex Bie et al. showed in Private GANs, Revisited that massive batch sizes allow a private GAN's discriminator to accurately distinguish the true data signal from injected noise, unlocking performance that rivals non-private baselines.
Overcoming Computational Bottlenecks
Processing millions of samples in a single training step is impossible on standard hardware due to memory constraints. To bypass these physical limitations, the community relies on architectural workarounds:
- Gradient Accumulation and Multi-Shard Parallelization: As implemented by Amir Feder et al. in Learning and Evaluating a Differentially Private Pre-trained Language Model, gradients are computed across multiple shards and accumulated over many sub-steps before the privacy noise is added and the weights are updated.
- Virtual Batching: Olivera Kotevska et al. show in Optimal Balance of Privacy and Utility with Differential Privacy Deep Learning Frameworks that decoupling the effective batch size from hardware memory via virtual batching provides the convergence stability needed for private deep learning frameworks.
The Interplay with Dataset and Model Scale
The necessity of large-batch scaling is tightly bound to the overall scale of the data and the model parameters. The authors of ViP: A Differentially Private Foundation Model for Computer Vision observe that scaling the total dataset size inherently improves the SNR under a fixed privacy budget, making massive data scaling a prerequisite for private training. Furthermore, Geeticka Chauhan et al. show in Training Large ASR Encoders with Differential Privacy that maintaining a constant SNR requires scaling the noise multiplier linearly with both the batch and dataset size, providing a predictable path for performance extrapolation on massive speech datasets.
However, model size introduces an additional complication. Rachid Guerraoui et al. point out in Differential Privacy and Byzantine Resilience in SGD: Do They Add Up? that privacy-preserving noise naturally scales with the number of parameters. To suppress this dimension-dependent noise, the batch size must be scaled up at a rate proportional to the square root of the parameter count, making large batches even more critical for large-scale architectures.
Nuances and Boundary Conditions
While extreme batch scaling is a dominant strategy, it is not a universal panacea, and it introduces unique trade-offs:
- The Privacy-Utility Tradeoff: As noted early on by Melissa Chase et al. in Private Collaborative Neural Network Learning, larger batches reduce the total number of gradient updates possible within a fixed privacy budget, requiring careful optimization. Eugene Bagdasarian et al. also caution in Differential Privacy Has Disparate Impact on Model Accuracy that mitigating noise via large batches can sometimes demand a larger privacy budget or more iterations to prevent a disparate impact on underrepresented subgroups.
- High-Noise Regimes: Ryan McKenna et al. challenge the "always larger" assumption in Scaling Laws for Differentially Private Language Models, showing that in exceptionally high-noise differential privacy regimes, smaller physical batch sizes can actually outperform larger ones due to fundamental hyperparameter trade-offs that differ from non-private optimization.
- Optimizer Dynamics: The interaction between large-batch noise and adaptive optimizers is delicate. Qiaoyue Tang et al. show in DP-Adam: Correcting DP Bias in Adam's Second Moment Estimation that DP noise biases Adam's second-moment estimation, a problem that behaves differently as gradient dynamics shift in large-scale training.
- Alternative Objectives: For non-decomposable objectives like contrastive loss, Weiwei Kong et al. demonstrate in Differentially private optimization for non-decomposable objective functions that bounding partial derivatives allows for large-batch training without letting the noise level scale linearly with the batch size. Meanwhile, alternative paradigms like forward-learning likelihood ratio proxies, explored by Mingqian Feng et al. in Forward Learning with Differential Privacy, exhibit unique stability properties under large batches compared to traditional gradient descent.
Key Foundational Work
Deep Learning with Differential Privacy (2016) — contributed 2 concept elements to this direction.
Private Collaborative Neural Network Learning (2017) — contributed 1 concept elements to this direction.