Simulating Large Batches via Gradient Accumulation

34 concepts32 papers1 proposal

Deep learning models often face a fundamental tension between optimization stability and hardware constraints. Large batch sizes are frequently necessary to stabilize training for architectures like the Transformer, as noted by Saunders et al., or to provide the global context required for ranking losses in image retrieval, as argued by Patel et al.. However, the memory required to store intermediate activations for these large batches often exceeds the physical capacity of modern GPUs.

Gradient accumulation—also referred to as delayed updates, batch subdivision, or virtual batching—resolves this by decoupling the logical batch size used for optimization from the physical batch size that fits in memory.

The Mechanics of Delayed Updates

The core intuition behind gradient accumulation lies in the linearity of the gradient operator. The gradient of a loss function calculated over a large batch is mathematically equivalent to the average of the gradients calculated over smaller sub-batches.

Instead of updating model weights after every forward and backward pass, the authors of CVIT-MT Systems for WAT-2018 explain that gradients can be summed over several sequential iterations. Only after a predefined number of steps is the accumulated gradient used to update the weights. This allows a single GPU to simulate the optimization dynamics of a much larger computational cluster. Bogoychev et al. demonstrate that this technique not only manages memory for large word embeddings in Neural Machine Translation (NMT) but also reduces communication frequency in asynchronous training, leading to significant speedups.

Decoupling Embeddings via Gradient Caching

While standard accumulation handles the memory of activations, certain tasks like metric learning and dense retrieval require comparing every sample in a batch to every other sample (e.g., for mAP or Recall@k). Storing all activations for a batch of thousands of high-resolution images is impossible even with accumulation.

To solve this, Revaud et al. and Patel et al. propose a multi-stage backpropagation strategy. In the first stage, the model performs a memory-efficient forward pass to compute only the final embeddings (descriptors) for the entire large batch, without storing intermediate activations. The loss and the gradients relative to these embeddings are then calculated. In the second stage, the model re-executes the forward and backward passes for small sub-batches to compute the actual weight updates, using the pre-calculated embedding gradients as a guide.

This "gradient caching" approach is further extended by Gupta et al. to handle Extreme Multi-label Classification (XMC), where the label set itself is too large to fit in memory. Similarly, Sourty utilizes gradient caching in the PyLate library to enable the training of memory-intensive multi-vector retrieval models on standard hardware.

Democratizing High-Performance Training

Beyond enabling new loss functions, these techniques serve as a vital tool for academic researchers with limited resources. Chen et al. show that by combining gradient accumulation with mixed-precision training, the HuBERT speech model—originally trained on massive industrial clusters—can be successfully pre-trained on a single node of eight GPUs.

The strategy is versatile across domains:

  • Object Detection: Sun et al. use batch subdivision to optimize YOLOv4 training on limited VRAM.
  • Video Captioning: Olivastri et al. employ accumulation to enable end-to-end training of combined CNN-RNN architectures.
  • Privacy: Yousefpour et al. implement "virtual batching" in the Opacus library to manage the memory-intensive per-sample gradients required for Differential Privacy.
  • Distributed Optimization: Aitchison et al. suggest accumulating squared gradients directly into the second-moment term of the Adam optimizer at the server level, further reducing memory overhead in large-scale systems.

By treating memory as a temporal rather than a spatial constraint, gradient accumulation allows researchers to scale the complexity of their models and the stability of their optimization without being strictly bound by the physical limits of their hardware.

Key Foundational Work

The RWTH Aachen University Supervised Machine Translation Systems for WMT 2018 (2018) — contributed 1 concept elements to this direction.
CVIT-MT Systems for WAT-2018 (2018) — contributed 1 concept elements to this direction.
Scaling Neural Machine Translation (2018) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_gradient_accumulation_6929aced,
  title = {gradient accumulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-accumulation/rdr_6929aced458f439d853b704dfda2b0b0},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-22},
}

This exact version (version 1, frozen text):

@misc{lacuna_gradient_accumulation_6929aced_v1,
  title = {gradient accumulation},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/gradient-accumulation/rdr_6929aced458f439d853b704dfda2b0b0/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-22},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake