In standard machine learning, choosing a batch size is often a matter of balancing hardware efficiency with generalization. Smaller batches are frequently preferred because they introduce "stochastic noise" that helps the model escape local minima. However, when we move to Differential Privacy (DP), the rules of optimization change fundamentally.
In the private regime, the primary obstacle to model utility is not the landscape of the loss function, but the explicit noise injected to protect individual data points. Recent research has converged on a singular, powerful intuition: to make private models work, we must scale batch sizes to levels that would be considered absurd in non-private training.
The Signal-to-Noise Problem
The standard algorithm for private training is DP-SGD. It differs from regular SGD in two key ways:
- Clipping: Each individual gradient in a batch is scaled so its norm does not exceed a threshold $C$.
- Perturbation: Random Gaussian noise is added to the aggregate batch gradient.
The magnitude of this noise is determined by the privacy budget ($\epsilon$) and the sensitivity of the operation (the clipping threshold $C$). Crucially, as Natalia Ponomareva and colleagues note in How to DP-fy ML, the amount of noise added is independent of the batch size.
This creates a clear mathematical lever. If you double the batch size, you are averaging the "signal" of twice as many data points, but you are adding the same amount of privacy noise. Consequently, increasing the batch size directly improves the Signal-to-Noise Ratio (SNR). As Xuechen Li and Florian Tramèr demonstrate in Large Language Models Can Be Strong Differentially Private Learners, this improved SNR is the primary driver of performance in private deep learning.
Why Subsampling Isn't a Silver Bullet
A common counter-argument in DP theory is privacy amplification by subsampling. The idea is that if you only look at a small random fraction of your data in each step, you gain a "privacy bonus," allowing you to add less noise.
However, Ossi Räisä, Joonas Jälkö, and Antti Honkela challenge this in Subsampling is not Magic. They argue that while small batches provide a privacy bonus, they also suffer from higher sampling variance. In the long-term limit of training, the privacy benefit of subsampling is mathematically offset by the need to take more steps. Large batches are superior because they minimize the variance introduced by the data itself while keeping the injected DP noise stable.
From Mini-batches to Mega-batches
To bridge the gap between private and non-private performance, researchers have pushed batch sizes to the extreme. Rohan Anil and the team at Google Research showed in Large-Scale Differentially Private BERT that training BERT-Large effectively required "mega-batches" of up to 2 million samples.
At this scale, the true gradient signal finally becomes dominant over the privacy noise. Similar results were found by Tom Sander and Yi Ma in Differentially Private Representation Learning via Image Captioning, where batch sizes exceeding one million were necessary to stabilize loss trajectories for vision models.
Mitigating Clipping Bias
Large batches don't just help with noise; they also help with bias. Because DP-SGD clips individual gradients, it can distort the direction of the aggregate update. Moritz Knolle and Alexander Ziller explain in Bias-Aware Minimisation that larger batches act as a buffer. They allow the aggregated gradient to more closely approximate the true population gradient, even when individual components are constrained by a maximum norm.
Practical Engineering for Massive Batches
Training with millions of samples per step is computationally daunting. Several techniques have emerged to make this feasible:
- Virtual Batching and Gradient Accumulation: Olivera Kotevska highlights that decoupling the effective batch size from hardware memory is essential. By accumulating gradients over multiple sub-steps before adding noise, practitioners can simulate massive batches on standard GPUs.
- Ghost Clipping: To avoid the memory overhead of calculating per-example gradients (required for DP clipping), researchers like Badih Ghazi utilize "ghost clipping" to compute the necessary norms more efficiently.
- Adaptive Optimizers: Standard SGD often fails at mega-batch scales. Harsh Mehta and Abhradeep Guha Thakurta found that layer-wise adaptive optimizers like LAMB are better at managing the specific SNR characteristics of noised gradients in Large Scale Transfer Learning for DP Image Classification.
A New Set of Scaling Laws
The shift toward massive batches suggests that private learning follows different scaling laws than traditional ML. Ryan McKenna and Da Yu propose in Scaling Laws for Differentially Private Language Models that we should think in terms of the noise-batch ratio. This ratio decouples training dynamics from specific privacy accounting, allowing researchers to predict how a model will perform across different privacy budgets by simply adjusting the batch size to maintain a target signal level.