Fewer Bits, More Crops: Unbiased Low-Precision Soft Labels for Distillation

Modern dataset distillation, which shrinks ImageNet into a few synthetic images per class, depends on a hidden cost. Each synthetic image is replayed under many random crops, and each crop needs its own stored teacher prediction: a soft label, meaning a full probability vector over all classes. These labels end up 30–40× larger than the images on ImageNet-1K and up to 200× larger on ImageNet-21K (LPQLD). Current fixes either store fewer crops or store fewer classes per crop. Storing fewer crops causes the student to learn from crops whose labels contradict the image as a whole, which HALD names local semantic drift. Storing fewer classes per crop keeps only the top few entries, which systematically biases the training signal (Sparse Logit Sampling).

Mixed-precision training has already solved a very similar problem. Tensors that are only consumed by a gradient computation can be stored at very low precision, as long as the rounding is randomized so that its expected value equals the true value. ActNN stores backward-pass activations at about 2 bits this way with almost no accuracy loss. We propose to treat soft labels the same way. The key fact is that the cross-entropy gradient with respect to the student's outputs is linear in the target probabilities. Randomized low-bit rounding of the stored probabilities therefore leaves the expected gradient exactly unchanged. That frees bytes, and the freed bytes can be spent on many more crops. Our claim is that, at a fixed storage budget, many coarse but unbiased labels beat a few precise ones. The reason is that the dominant failure under compression is lack of crop coverage, not lack of label precision.

Core Claim

Let $z$ be the student's logits for a crop, $T$ the distillation temperature, and $q \in \mathbb{R}^C$ the teacher's tempered probability vector over $C$ classes. For the soft cross-entropy loss $\ell = -\sum_c q_c \log \mathrm{softmax}(z/T)_c$, the gradient is

$$ \nabla_z \ell = \tfrac{1}{T}\Big(\big(\textstyle\sum_c q_c\big)\,\mathrm{softmax}(z/T) - q\Big). $$

This expression is affine in $q$. It holds even when $q$ does not sum exactly to one.

Now let $Q(q)$ be a stochastic quantizer. Each entry is rounded to one of its two neighbouring levels on a $b$-bit grid with step $\Delta$, with probabilities chosen so that $\mathbb{E}[Q(q)] = q$. The expected gradient is then unchanged. The added per-sample variance is at most

$$ \frac{C\,\Delta^2}{4T^2}, $$

because each coordinate's rounding error lies within one step $\Delta$. The rounding noise is drawn once at storage time, so for a fixed stored dataset it is a frozen perturbation. That perturbation is independent and zero-mean across crops and images, so it averages out over a minibatch. Top-$K$ truncation behaves differently: it always zeroes or flattens the same tail, so its error never averages out.

This gives a concrete exchange rate between bits and crops, and an empirical prediction. On ImageNet-1K at a fixed byte budget, the best operating point will store several times more crops than LPQLD or HALD do, with only 2–4 bits per tail class.

Two things are deliberately excluded. We do not quantize logits, as LPQLD does. A softmax over quantized top-$k$ logits is nonlinear in the stored values, which is why LPQLD needs temperature calibration by grid search. We also do not pursue the "FP16 underflows the soft-label tail" concern. Standard practice already computes the loss operator in FP32, so we only verify this in a cheap sanity run.

Research Program

The first component is the label codec. It works in probability space at the training temperature. The top $m$ classes per crop are stored as index plus 8-bit value. The remaining $C-m$ classes are handled by an unbiased compressor of the residual mass, in one of two variants. The first variant is dense $b$-bit stochastic rounding with a per-crop scale, borrowing ActNN's per-group scaling. The second is importance-sampled sparse entries as in Sparse Logit Sampling. For a given byte count, the codec picks whichever has lower variance. At $C = 1{,}000$ the sparse variant should win at very low budgets. At $C \approx 21{,}000$, dense 1–2 bit coding of the tail may win, and we will measure where the crossover falls.

The second component is the budget allocator. For a total byte budget $B$, it chooses the number of crops per image, $m$, and $b$, and labels are stored using the crop-coordinate replay of FKD. Following ActNN's sensitivity-based bit allocation, a cheap pilot run on Tiny-ImageNet measures two quantities: how much gradient variance each extra bit removes, and how much each extra crop adds in label diversity, measured as disagreement between a crop's teacher label and the image-level label. The allocator then picks the point that minimizes total gradient variance.

The third component is loading. Labels stream to the GPU in packed form and are dequantized inside the loss kernel, which keeps I/O proportional to the compressed size.

One limitation is real. LPQLD's dynamic temperature annealing needs logits, while our probabilities are tied to one temperature. We will either store at a single temperature or store two tempered copies within the same budget, and report the cost of losing annealing.

Baselines

The naive reference is SRe²L-style relabeling. It stores full-precision soft labels for every crop and is the uncompressed upper bound. It reached 60.8% at IPC 50 on ImageNet-1K (SRe²L). IPC means images per class. A second naive point is the same pipeline with labels simply cast to FP16 or BF16, which gives a free 2× reduction.

The strongest competitor at aggressive budgets is HALD. It uses a soft–hard–soft schedule with heavily smoothed hard labels and reports 42.7% ImageNet-1K accuracy with 285 MB of soft-label storage, outperforming LPLD by 9.0%. We will run it from the official code (Jiacheng8/HALD) on the same SRe²L and RDED synthetic images, with the same student and epochs. Beating HALD means exceeding 42.7% at 285 MB, or matching it at clearly fewer bytes. Our codec is compatible with HALD's schedule, so we also test HALD's schedule fed with our labels. If that combination is best, the honest framing is that our method complements HALD rather than replacing it.

At moderate budgets the competitors are LPQLD (78× reduction on ImageNet-1K, 500× on ImageNet-21K-P) and SCORE. SCORE reports 55.9 GB reduced to 1.9 GB at IPC 50 while retaining 95.7% relative accuracy, and a drop of only 2.7% under 30× compression at IPC 50. The file gives no absolute LPQLD accuracy, only relative gains; there is no reported result found on Lacuna for LPQLD's absolute top-1 at matched budgets. We will reproduce LPQLD ourselves at its byte count, and compare to SCORE at 1.9 GB. LPQLD keeps its full calibration grid search, and our allocator pilot is capped at the same GPU-hour budget.

For the full-data distillation setting, FKD is the baseline. Its ResNet-50 reached 80.1% with 13–22 GB of top-$K$ labels, and we ask whether that accuracy is reachable at a fraction of the bytes.

Three ablations isolate the key ingredient. The first uses identical bits but deterministic rounding. The second uses identical bytes but spends them on fewer, higher-precision crops. The third quantizes logits instead of probabilities.

Why not just…?

Why not just store soft labels in FP8 or INT4 with ordinary rounding? At 8 bits this probably works, and we expect it to roughly match full precision. That is why it appears as an ablation rather than being ignored. The proposal's advantage appears below about 4 bits. There, deterministic rounding sends most tail probabilities (around $10^{-4}$ at $C = 1{,}000$) to zero, which recreates top-$K$ bias. If deterministic 2-bit rounding matches stochastic 2-bit rounding at equal bytes, the unbiasedness argument is empirically irrelevant, and we will say so. The crops-versus-bits trade-off would still stand as a finding.

Why not just use top-$K$ truncation, as FKD and LPQLD do? Sparse Logit Sampling shows that deterministic top-$K$ biases the gradient. LPQLD needs temperature alignment and annealing to avoid collapse under top-$k$ logits. The decisive check is a direct one. We measure the mean gradient on student logits under each codec against the full-label gradient over a fixed minibatch set, and report bias norm separately from variance.

Why not just use HALD's hard-label anchoring and stop storing more? This is the real threat, because HALD already attacks drift at 285 MB. HALD fixes drift by replacing crop-specific information with crop-invariant labels. We instead keep crop-specific information and pay for it in bits. The matched-budget comparison, together with the HALD-plus-our-labels hybrid, settles whether that information is worth recovering.

Why not just recompute the teacher online? Online recomputation removes storage entirely, but FKD reports that its cached approach is 3–5× faster than vanilla KD. Online recomputation also forfeits the portability that motivates distilled datasets. We report it only as the compute-heavy reference.

Proposed Experiments

Fast sweeps run on Tiny-ImageNet and ImageWoof. The main results use ImageNet-1K at IPC 10 and 50, with a ResNet-18 student on SRe²L and RDED images, at three byte budgets: 285 MB, LPQLD's 78× point, and 1.9 GB. A scaling test runs on ImageNet-21K-P, where large $C$ should favour dense low-bit tails over sparse sampling. A full-data check uses FKD's ResNet-50 setting.

Metrics are top-1 accuracy, label bytes, data-loading throughput, and measured gradient bias and variance. Each budget is run with three seeds.

The decisive experiment is the equal-byte frontier: accuracy as a function of crops per image, with bits adjusted to keep total bytes constant. The hypothesis is falsified if accuracy peaks at LPQLD/HALD-like crop counts with high precision. It is confirmed if the peak sits at several times more crops with 2–4 bit tails, and if stochastic rounding beats deterministic rounding at that peak.