Overcoming the Soft Label Storage Bottleneck

42 concepts25 papers1 proposal

Knowledge distillation and dataset distillation are designed to make machine learning more efficient. Knowledge distillation shrinks massive models into lightweight "students," while dataset distillation condenses millions of training images into a handful of highly informative synthetic samples.

However, these techniques suffer from a hidden paradox: the soft label storage bottleneck. To transfer rich "dark knowledge" from a teacher model to a student, frameworks rely on soft labels—dense probability distributions across every possible output class or vocabulary token. While a standard "hard" label is just a single integer, a soft label is a high-dimensional floating-point vector.

When scaling to large datasets, many classes, or massive vocabularies, storing these soft labels creates a massive "storage tax." The metadata required to train the compressed models can easily exceed the size of the original uncompressed data, creating a severe logistical bottleneck.


The Root of the Inflation

To understand why soft labels bloat storage, consider how the data scales. If a dataset has $N$ samples and $C$ target classes, a hard label dataset requires storing exactly $N$ integers. A soft label dataset, however, requires storing $N \times C$ floating-point values.

This linear scaling with respect to the label space cardinality ($C$) becomes prohibitive in two major domains:

  1. Large-Scale Classification: For a dataset like ImageNet-1K ($C=1,000$), storing full floating-point distributions for millions of samples and their various cropped augmentations requires terabytes of disk space, as noted by Eric P. Xing et al. in their work on Fast Knowledge Distillation (FKD).
  2. Language Modeling: In Large Language Models (LLMs), the vocabulary size ($V$) routinely spans from 32,000 to over 100,000 tokens. As shown by Hao Peng et al. in Pre-training Distillation for Large Language Models, caching full teacher distributions across trillions of tokens during the pre-training phase demands petabyte-scale storage, making offline distillation logistically impractical.

Furthermore, high-performance dataset distillation often requires unique soft labels for every augmented version of a synthetic image or across every training epoch to prevent student models from overfitting. As highlighted by Ruonan Yu et al., this causes the auxiliary soft labels to consume orders of magnitude more space than the synthetic images themselves, defeating the primary purpose of data condensation.

To make distillation practical at scale, researchers have developed four core strategies to alleviate this storage bottleneck.


Strategy 1: Sparsification and Smart Sampling

The intuitive solution to high-dimensional vectors is to throw away the values close to zero. In a 1,000-class problem, a teacher model might concentrate 99.9% of its probability mass on the top 3 classes, leaving a long tail of near-zero logits.

However, simply truncating the distribution to a standard "Top-K" representation introduces statistical bias and discards the subtle inter-class relationships that make distillation effective. To bypass this, Anshumann et al. developed Random Sampling Knowledge Distillation (RS-KD). Instead of deterministically keeping the largest logits, their framework uses importance sampling to select a sparse subset of logits. This preserves an unbiased estimate of the full distribution gradient, reducing LLM logit storage by 25x while accelerating training.

Full Logits (Size V)   ───>   [ Importance Sampling ]   ───>   Sparse Logits (0.01% Size)
(Petabytes of space)                                            (Identical gradient properties)

In the computer vision space, Lingao Xiao et al. introduced Label Pruning for Large-scale Distillation (LPLD). By systematically pruning redundant label metadata and shifting the focus toward class-wise supervision, they achieved a 40x compression of soft labels on ImageNet-1K while simultaneously boosting student accuracy by freeing up capacity for synthetic image diversity.


Strategy 2: Collapsing the Temporal Dimension via Static Labeling

During dataset distillation, a common practice is to save dynamic soft labels that change across training epochs or vary with every random data augmentation. This temporal tracking multiplies the storage footprint by the number of training steps.

To break this dependency, researchers have shifted toward static labeling strategies. In the INFER framework proposed by Xin Zhang et al., the authors decouple the labels from the training timeline by averaging the teacher model's outputs into a single, fixed probability distribution per synthetic instance. By utilizing a Universal Feature Compensator to blend features across classes rather than storing epoch-specific labels, they slashed the soft label storage footprint on ImageNet-1K by 99.3% without sacrificing final model accuracy.


Strategy 3: On-the-Fly Label Generation

Instead of compressing or caching soft labels on disk, what if we could eliminate label storage entirely by generating them mathematically on-the-fly?

This is the core intuition behind the HeLlO (Heavy Labels Out) framework introduced by Ruonan Yu et al.. Instead of saving massive arrays of teacher-generated soft labels for every synthetic image and its augmentations, the authors train a lightweight, CLIP-based image-to-label projector.

[ Synthetic Image ] ───> [ Lightweight Projector (CLIP) ] ───> On-the-Fly Soft Label

During student training, this compact projector takes the synthetic image as input and instantly predicts its corresponding soft label. This bypasses the need for disk storage entirely, reducing the label storage requirement to a staggering 0.003% of traditional methods.

Similarly, in the context of Non-Critical Region Refinement (NRR-DD), Minh-Tuan Tran et al. introduced a distance-based label compression technique. By anchoring soft labels to geometric distances within a shared embedding space rather than storing raw probability vectors, they achieved a 500x reduction in storage overhead for large-scale datasets.


Strategy 4: Quantization and Delta Coding

When distillation occurs in distributed environments—such as Federated Distillation (FD), where edge devices exchange soft labels on a public dataset instead of sharing heavy model weights—the storage bottleneck transforms into a communication bottleneck. Transmitting uncompressed, high-precision floating-point vectors across a network quickly saturates available bandwidth.

To resolve this, Felix Sattler et al. introduced Compressed Federated Distillation (CFD). Their approach applies two bit-level compression mechanisms directly to the soft labels:

  • Quantization: Mapping high-precision 32-bit floating-point probabilities to low-bitwidth representations (e.g., 8-bit or fewer).
  • Delta Coding: Transmitting only the changes (deltas) in the soft labels between successive communication rounds, rather than re-sending the entire distribution.

By compressing the bit-width and eliminating temporal redundancy, CFD reduces communication overhead by up to four orders of magnitude compared to traditional federated averaging, making distributed distillation viable over constrained networks.


Summary

The soft label storage bottleneck highlights a critical tension in machine learning efficiency: compressing a model or a dataset often creates a massive footprint of auxiliary metadata. By moving away from naive, full-precision logit caching and adopting techniques like importance sampling, static labeling, on-the-fly projection, and quantization, modern distillation frameworks can successfully scale to massive benchmarks like ImageNet-1K and multi-billion token LLMs without succumbing to storage bloat.

Key Foundational Work

Ensemble Distillation for Structured Prediction: Calibrated, Accurate, Fast - Choose Three (2020) — contributed 1 concept elements to this direction.
Distilling Knowledge Learned in BERT for Text Generation (2020) — contributed 1 concept elements to this direction.
CFD: Communication-Efficient Federated Distillation via Soft-Label Quantization and Delta Coding (2021) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_soft_label_storage_overhead_193f266f,
  title = {soft label storage overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/soft-label-storage-overhead/rdr_193f266f0f4245c3b7af2b436f7873df},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 1, frozen text):

@misc{lacuna_soft_label_storage_overhead_193f266f_v1,
  title = {soft label storage overhead},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/soft-label-storage-overhead/rdr_193f266f0f4245c3b7af2b436f7873df/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-19},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake