Large scientific datasets are read in contiguous chunks because random reads are too slow. Two examples are single-cell omics files grouped by experimental plate and satellite archives grouped by geographic tile. The problem is that neighbouring records are alike: they come from the same plate, patient or region. A minibatch built from a few chunks is then worth fewer independent examples than its size suggests. Today the chunk size and the in-memory shuffle buffer are set by a fixed rule of thumb or by expensive tuning runs. We propose to set them from a measured quantity instead. That quantity is how much of the per-example gradient variation is shared within a chunk rather than across chunks. We would estimate it with the variance-components machinery of random-effects meta-analysis, a statistical model that splits variation in study results into within-study noise and genuine between-study differences.
The payoff is a loader that reads the largest chunks the data can tolerate. When chunks are nearly exchangeable, it reads big contiguous blocks and gets close to sequential disk speed. When chunks differ sharply, as with Tahoe-100M plates, it mixes more aggressively. Both choices follow from one formula that can be checked directly against measured gradient noise.
Core Claim
Random-effects meta-analysis models each study's true effect as a global mean plus a study-specific deviation with variance $\tau^2$, observed with within-study noise $\sigma^2$ (Random Effects Models for Statistical Meta Analysis). We apply the same decomposition to per-example gradients. The "study" is a contiguous storage block, nested inside a plate or tile.
The intraclass correlation $\rho = \tau^2/(\tau^2+\sigma^2)$ is the fraction of gradient variance explained by block membership. Classical cluster-sampling theory then gives the inflation of minibatch gradient variance relative to true random sampling. Suppose blocks of size $b$ are loaded into a buffer holding $f$ minibatches of size $B$ and reshuffled, as in scDataset. Each example then shares its block with about $(b-1)(B-1)/(fB-1) \approx b/f$ other examples in its minibatch. The resulting variance inflation, or "design effect", is
$$D(b,f) \approx 1 + \frac{b}{f}\,\rho,$$
where the approximation counts expected same-block pairs in a batch drawn uniformly from the buffer. A term for same-plate but different-block pairs, with its own correlation, is added in the nested version.
The effective batch size is $B/D$. We choose $(b,f)$ to maximize measured read throughput subject to $D \le D^\ast$ and to a buffer memory limit $fB \le M$.
This recasts scDataset's rule $b \le f/2$ as the special case $D \le 1+\rho/2$ that ignores what $\rho$ actually is. Our claim is that $\rho$ varies by an order of magnitude across datasets, block definitions and phases of training. A $\rho$-aware choice should therefore dominate the fixed rule on the throughput-versus-accuracy frontier.
One favourable twist: meta-analysis struggles to estimate $\tau^2$ when there are few studies. Here there are thousands of blocks per epoch, so standard estimators such as DerSimonian–Laird or restricted maximum likelihood are well posed.
Relation to Prior Work
The motivating evidence comes from scDataset. It shows that sequential streaming of Tahoe-100M gives near-chance accuracy and loss spikes at plate boundaries. It also shows that quasi-random block sampling recovers true-random accuracy at more than 200× the throughput of AnnLoader. However, it sets $b$ and $f$ by hand and names automated tuning as open. We keep its sampler unchanged and replace only the rule for choosing its parameters.
BigEarthNet v2 documents spatial autocorrelation among nearby patches, which supplies a second, independent cluster structure. Harmonizing Multi-Site Data Pipelines to Mitigate Batch Effects supports treating site or plate differences as a mixture of real signal and technical noise that cannot simply be preprocessed away. That is exactly the situation a random-effects variance term is meant to describe.
Systems work constrains the design. Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data tunes worker counts, compression and tile alignment with Bayesian optimization, purely at the systems level. It shows that tile-aligned reads matter, but it never asks what the chosen read granularity does to the statistics of each minibatch. Our method takes such a tuned systems configuration as given and adds the statistical constraint on top. Profiling and Improving the PyTorch Dataloader for high-latency Storage and Clairvoyant prefetching for distributed machine learning I/O hide latency through concurrency and prediction of future accesses. Both are complementary to our approach and would be kept.
The closest statistical relatives are the data-reuse papers. Faster Neural Network Training with Data Echoing and Stochastic Optimization with Laggard Data Pipelines study repeated batches, and Augment Your Batch studies replicated instances within a batch. They concern correlation created by reusing the same examples at a fixed factor. We concern correlation that already exists among distinct stored examples, and we measure it rather than fixing it.
We deliberately do not pursue the more obvious bridge suggested by the research file: reweighting each batch's or worker's gradient by $1/(\sigma_i^2+\tau^2)$. Reasons are given below.
Research Program
The first component is an online estimator. Every few hundred steps, the loader draws a small probe of about 64 blocks with 32 examples each. It computes per-example gradients of the final layer, which is cheap, and projects them onto a handful of random directions. It then fits a nested random-effects model with components for block, plate or tile, and residual. This yields estimates of $\rho_{\text{block}}$ and $\rho_{\text{plate}}$ with confidence intervals.
The second component is a throughput model. We take one short timing sweep of read time against $b$ on the actual storage tier (local NVMe, a parallel filesystem, S3). We fit it as a fixed per-request cost plus a bandwidth term.
The third component is the controller. At each re-estimate it solves the small constrained problem above and changes $b$ and $f$ at epoch or buffer boundaries.
The analysis part tests whether $D(b,f)$ predicts two things. The first is measured minibatch gradient variance relative to true random sampling. The second, more importantly, is the accuracy penalty relative to true random sampling at matched steps. We also track how $\rho$ drifts during training, since the gradient correlation within a plate may grow once easy global features are learned. This drift is why the estimator runs online rather than once.
Baselines
The naive reference points bracket the problem. Sequential streaming is fastest and, per scDataset, gives near-chance accuracy on Tahoe-100M cell-line, drug and mechanism-of-action labels. True random sampling sets the accuracy ceiling, and on 100M cells it is prohibitively slow; scDataset estimates nearly two months per epoch. Where full random runs are infeasible, we run them on a subsample.
The strongest existing method is scDataset with its $b \le f/2$ rule. We give it its best $(b,f)$ from a grid search, using a tuning budget equal to what our probe plus timing sweep costs, measured in GPU-hours. Its reported result is matching true-random predictive performance at 204× AnnLoader throughput. Its absolute Tahoe accuracies do not appear in the file: no reported result found on Lacuna for Tahoe-100M classification accuracy values.
The second strong competitor is Bayesian optimization of loader parameters in the style of Optimizing Cloud-to-GPU Throughput, extended to include $b$ and $f$. It optimizes validation accuracy within a fixed wall-clock budget. That paper reports up to 20× remote throughput and 6–15% validation IoU gains within identical time budgets over default settings. On reBEN, the reference accuracy is a ResNet-101 macro average precision of 70.93% with fused Sentinel-1 and Sentinel-2 input (BigEarthNet v2).
Beating these baselines means one of two things. Either we reach the same accuracy (within the seed-to-seed standard deviation of true random sampling) in at least 1.3× less wall-clock time, or we reach higher accuracy at equal wall-clock time. In both cases the tuning cost is included in the time.
Three ablations isolate the key ingredient. The first uses a single $\rho$ estimated at step zero instead of online estimates. The second uses $\rho$ computed from label co-occurrence rather than from gradients. The third drops the plate-level component and keeps only the block-level one.
Why not just…?
Why not just tune $b$ and $f$ with Bayesian optimization? This is the real threat. On a single fixed dataset and storage tier, enough tuning runs will find a good configuration. Our argument is cost and transfer. Each tuning evaluation is a partial training run, whereas our choice comes from a probe lasting minutes. The experiment above gives both methods the same GPU-hour budget. If Bayesian optimization matches us at equal budget on all three datasets, the contribution shrinks to an interpretability result: the design effect explains the configuration the optimizer found. We would report that outcome honestly.
Why not just convert to a fast random-access format? FFCV shows large gains from its custom format. However, the Earth-observation paper notes that public archives often have fixed formats. For Tahoe-100M or petabyte satellite archives, a full conversion and local copy is itself the bottleneck. Where conversion is feasible, true random sampling becomes cheap and our method should choose $b \approx 1$. We include this setting as a sanity check that the controller does not hurt.
Why not just reweight gradients by $1/(\sigma_i^2+\tau^2)$? Inverse-variance weights are the textbook meta-analytic move, but they answer a different question: how to combine noisy estimates of one scalar. Clustering inside a minibatch reduces the information in that batch. Reweighting batches cannot restore that information; it can only reallocate step sizes, which learning-rate schedules and adaptive optimizers already do. We run it as an extra comparison on top of scDataset's sampler. We expect it to fail to remove plate-boundary loss spikes.
Why not just raise the batch size by $D$? Scaling $B$ by the design effect restores effective sample size. It also multiplies memory and changes the optimization dynamics that a large batch brings, which the laggard-pipeline analysis shows interact with the variance regime. We include "scDataset rule with batch size multiplied by $D$" as a baseline. If it matches our method, the practical lesson becomes "measure $\rho$, then pick the batch size". That lesson would still rest on our estimator.
Proposed Experiments
We use three datasets with different cluster structure. The first is Tahoe-100M in AnnData/HDF5 through scDataset, with blocks nested in plates and targets of cell line, drug and mechanism of action. The second is reBEN, with patches ordered spatially within countries and read as LMDB and as cloud GeoTIFF tiles. The third is ImageNet stored in class-sorted shards, a known worst case with very high $\rho$ that we can create deliberately.
Each is run on local NVMe and on a high-latency tier (S3 or a parallel filesystem). We measure samples per second, GPU utilization, wall-clock time to target accuracy, and final accuracy or average precision over three seeds.
Two checks are decisive. The first is calibration: across a grid of $(b,f)$, predicted $D$ against measured gradient-variance ratios should fall close to the identity line. The accuracy gap to true random sampling should also be ordered by $D$ rather than by $b/f$ alone. If configurations with equal $b/f$ but different measured $\rho$ show the same accuracy gap, the core claim is false.
The second is the frontier. On data with low $\rho$, the controller should choose blocks well beyond the $b \le f/2$ limit and gain throughput with no accuracy loss. On class-sorted ImageNet, it should tighten mixing where the fixed rule degrades.
References
- scDataset: Scalable Data Loading for Deep Learning on Large-Scale Single-Cell Omics
- BigEarthNet v2
- Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data
- Profiling and Improving the PyTorch Dataloader for high-latency Storage: A Technical Report
- Clairvoyant Prefetching for Distributed Machine Learning I/O
- Faster Neural Network Training with Data Echoing
- Stochastic Optimization with Laggard Data Pipelines
- Augment Your Batch: Improving Generalization Through Instance Repetition
- FFCV: Accelerating Training by Removing Data Bottlenecks