Spending Disk Reads on the Tail: Two-Tier Autoencoder Storage for Out-of-Core Training

When a training set is larger than a machine's memory, a common workaround is memory mapping. The operating system treats the file on disk as if it were an array in memory and loads each piece only when the program touches it. This works surprisingly well, but it moves the bottleneck to the disk. The clearest measurement in our material is from M3: Scaling Up Machine Learning via Memory Mapping. It trained logistic regression and $k$-means on 190 GB of handwritten digits on one PC and saw 100% disk utilization while the CPU sat at about 13%. The processor was mostly waiting for data.

An autoencoder is a neural network trained to squeeze each input into a short code and rebuild it from that code. It could turn that idle compute into fewer bytes read. The known catch is that autoencoders treat rare, unusual inputs as noise. Rare digit classes, extreme rainfall and odd sensor readings are exactly what gets blurred away. We propose a two-tier memory-mapped store. The first tier holds short autoencoder codes for every sample, is small enough to stay cached in memory, and is decoded on the fly. The second tier, kept on disk, holds the exact residual (the input minus its reconstruction). A cheap rule decides, per sample, whether to fetch that residual. The disk budget is therefore spent almost entirely on the few samples the autoencoder reconstructs badly. The claim is that this recovers most of the throughput of training on compressed codes while keeping accuracy on rare cases close to training on raw data.

Core Claim

Let $N$ be the number of samples, $D$ the bytes per raw sample, $c$ the bytes per code, $r$ the bytes per stored residual, and $p$ the fraction of samples whose residual is fetched in an epoch. Disk traffic per epoch then falls from $ND$ to roughly

$$N\,(c + p\,r),$$

with the code tier served from cache after the first pass.

Whenever $p$ is small, the speedup over raw memory mapping is therefore bounded by $D/(c+pr)$. The bound follows directly from M3's observation that the workload is limited by disk reads rather than computation.

What is new is not compressing data. Neither is the residual coding itself, which comes from the error-bounded sensor compression of Rate-Distortion Balanced Data Compression for Wireless Sensor Networks. What is new is making residual retrieval selective and driven by paging. The fetch rule uses a per-sample score kept in memory: the stored reconstruction error, optionally combined with the downstream model's current loss on that sample's reconstruction. The rule is tuned so that page faults concentrate on the tail.

We claim three things. First, the fetch rule achieves $p$ of a few percent at a fixed accuracy target. Second, errors-by-rarity are what it fixes. Third, a non-linear autoencoder buys a smaller $p$ than a linear projection at equal code size. The third claim is the one most likely to fail, and the experiments are designed to test it directly.

Relation to Prior Work

Three systems motivate the storage side.

  • M3 supplies the out-of-core regime and the evidence that disk, not compute, is the limit. It covers only linear models and $k$-means, and never reduces the bytes read.
  • ColBERT-Serve keeps dense token embeddings memory-mapped. It limits page faults to about 200 candidates chosen by a sparse lexical filter, cutting Wikipedia index memory from 98.3 GB to 8.2 GB. Our design has the same shape: a cheap in-memory stage decides which disk pages to touch. The difference is that our filter is learned per sample for training, not text-based for retrieval.
  • RainBench introduced PyRain, a memory-mapped loader reporting up to a 60.3× speedup over NetCDF loaders. We use PyRain as the raw-data reference. RainBench also documents the blurring that mean-squared-error models produce on precipitation, which motivates our focus on the tail.

Two works constrain the design.

  • Non-Volatile Memory Accelerated GMRA ran a multiscale linear dimensionality reduction over memory-mapped persistent memory with only a 1.04–1.14× slowdown versus DRAM. This is a warning that linear methods on fast storage are a strong, cheap competitor.
  • Privateer 2.0 replaced kernel paging with user-space paging that adds block compression and deduplication. It reports up to 300× storage reduction on snapshots. It is the lossless alternative our lossy-plus-residual tier must beat on real sensor and image data, which has little exact repetition.

Learning Compressed Sentence Representations for On-Device Text Processing supplies evidence on the code tier. Its autoencoder binarized 4096-dimensional vectors with 98.5% storage savings, staying within about 2% of original accuracy. It also found random projections competitive at short code lengths, another reason the linear ablation is mandatory.

Research Program

Building the store. We will write the two-tier store as memory-mapped arrays: codes in one contiguous file and residuals in a second. Residuals are grouped in fixed-size blocks so that one fetch costs one or a few pages. Per-sample scores stay in a small in-memory table. Loading uses standard memory-mapped containers such as the Arrow-backed format of Datasets and TorchRL's TensorDict (TorchRL).

Training the autoencoder. The autoencoder is trained on a streamed subsample of the data in a single pass. Its training time, and the one-time encoding pass over the full dataset, are charged to our method in every timing comparison.

The fetch rule. We will study three variants of increasing cost:

  • a fixed threshold on stored reconstruction error;
  • a loss-aware variant that fetches when the downstream model's loss on the reconstruction exceeds a running quantile;
  • a budgeted variant that fetches the top-$p$ fraction of each mini-batch by score.

We also analyze when the loss-aware rule can starve samples. A sample misclassified from its reconstruction is fetched and corrected, but a sample confidently wrong from its reconstruction may never be fetched. We will measure this directly by auditing a random slice with full residuals.

Baselines

Raw memory mapping. This is the do-nothing reference: plain memory mapping of the raw array, in the style of M3 on Infimnist and PyRain on RainBench. It sets the accuracy ceiling and the throughput floor. The strongest reported numbers here are system-level only: M3 ran logistic regression about 30% faster than an 8-instance Spark cluster while at 100% disk and roughly 13% CPU utilization, and PyRain reached up to 60.3× over NetCDF loaders. We found no reported downstream accuracy on Lacuna for M3's Infimnist logistic regression, nor a forecasting skill score for PyRain-trained models in this file. We will therefore measure both ourselves on identical hardware.

Lossless chunked compression. This is the strongest system competitor: block compression in the spirit of Privateer 2.0, and Zarr-style chunking as used by TerraMesh. It is exact, so any accuracy loss we show counts against us. We run it on the same disk and the same number of loader workers, with the compressor level tuned for throughput.

Codes-only training. This baseline trains on autoencoder reconstructions or codes with no residual tier. It measures the throughput ceiling and exposes tail neglect.

Linear two-tier ablation. This is the ablation that isolates our key ingredient: the identical two-tier store and fetch rule with PCA or random-projection codes instead of the autoencoder, at matched code bytes.

Random fetch ablation. This keeps the autoencoder but fetches residuals for a random $p$ fraction. It isolates the value of score-driven fetching.

Tuning and winning criteria. Every method gets the same hyperparameter budget and the same downstream model. Our method beats raw mapping if it matches raw accuracy on the rare subset within one point while cutting wall-clock time per epoch, including amortized encoding cost, by at least 2×. It beats lossless compression if it is faster at equal rare-subset accuracy.

Why not just…?

Why not just use PCA or random projections? This is the real threat. GMRA on persistent memory and the random-projection results in the sentence-compression paper both suggest linear codes may suffice. The sensor-compression paper found autoencoders beat PCA and DCT on reconstruction error, but reconstruction error is not the quantity we care about. The linear two-tier ablation settles this with fetch fraction $p$ at equal accuracy as the metric. If PCA reaches the same $p$, the contribution reduces to the selective residual store with a linear coder, and we will report it that way.

Why not just compress losslessly? Lossless coders exploit repeated bit patterns. Privateer's 300× gains came from incremental snapshots with heavy duplication. Continuous imagery and weather fields rarely repeat exactly, so we expect much lower ratios there. The comparison measures this directly rather than assuming it, and a combined variant that losslessly compresses our residual tier is included.

Why not just add machines or memory? M3 shows a single memory-mapped machine already beating an 8-instance Spark cluster at 190 GB. Our question is how far that single machine can go before scaling out. Buying memory remains a valid answer below some dataset size, and we will report the crossover size.

Why not just train on frozen pretrained features? This works when a good pretrained encoder exists and the downstream model never needs raw inputs. The residual tier exists for the opposite case: models that must see full-fidelity rare inputs. The codes-only baseline serves as a proxy for this alternative.

Proposed Experiments

Infimnist. We scale Infimnist to M3's 190 GB and train logistic regression, $k$-means and a small convolutional network. To create a long tail, we downsample two digit classes to 1% frequency. We report epoch time, bytes read, CPU utilization, overall accuracy and rare-class accuracy.

RainBench. Through PyRain, we train RainBench's convolutional forecasting model and report error separately on the heaviest-precipitation percentile. This is where tail neglect should hurt codes-only training most.

TerraMesh. As a stretch goal, a TerraMesh subset tests multimodal data where grid chunking already exists.

Decisive checks. The first check is whether score-driven fetching beats random fetching on rare-subset accuracy at equal $p$; if not, selectivity adds nothing. The second is whether the autoencoder beats linear codes on $p$ at matched code bytes; if not, the neural component is unjustified. The third is whether measured speedup tracks the $D/(c+pr)$ bound. A shortfall would mean decoding, not disk, has become the new bottleneck.