A Dataset is Worth 1 MB

Elad Kimchi Shoshani, Leeyam Gabay, Yedid Hoshen

In the world of distributed machine learning, we usually face a frustrating trade-off. If you want to train a model on a remote client (like an autonomous vehicle or a medical device), you either have to send a massive dataset of raw pixels—which is slow and expensive—or send pre-trained model weights. However, model weights are often non-portable across different hardware or software frameworks (e.g., moving from a PyTorch server to a JAX-based medical sensor).

A new paper, "A Dataset is Worth 1 MB," introduces a clever workaround called PLADA (Pseudo-Labels as Data). Instead of sending images or weights, the authors suggest we should just send the labels.

TL;DR

PLADA eliminates the need to transmit image pixels by leveraging a large, unlabeled "reference dataset" already stored on the client. By transmitting only highly compressed pseudo-labels for a semantically relevant subset of these images, the method enables high-accuracy task transfer with a payload of less than 1 MB.

What Problem Does This Paper Address?

Transmitting datasets is a massive bottleneck in "dataset serving." For a robot on Mars or a sensor at the bottom of the ocean, bandwidth might be as low as 500 bits per second. Sending a standard 1 GB dataset would take months.

Existing solutions like dataset distillation try to compress datasets into a few "synthetic" images, but these methods struggle to scale to high resolutions and often result in files that are still several megabytes or larger. Furthermore, they don't solve the "heterogeneity problem": different clients need raw data to optimize models for their specific local hardware.

Methods

The core idea of PLADA is to assume that the server and the client both have a copy of a massive, unlabeled reference dataset (like ImageNet-21K).

  1. Teacher Training: The server trains a "teacher" model on the actual target task (e.g., identifying specific bird species).
  2. Pseudo-Labeling: The server runs this teacher model over the reference dataset to generate "pseudo-labels."
  3. Energy-Based Pruning: Most images in a generic reference set (like ImageNet) are irrelevant to a specific task (like medical imaging). The server uses Energy-based Out-of-Distribution (OOD) scores to identify which reference images the teacher is most confident about.
  4. Safety-Net Filtering: To prevent the model from only learning "easy" classes, the authors introduce a quota system that ensures even "hard" or under-represented classes are included in the transmitted subset.
  5. Compression: The server sends a tiny payload containing the indices of the selected images and their corresponding labels, compressed using Zstandard (Zstd).
  6. Local Training: The client reconstructs the labeled subset from its local storage and trains its own "student" model from scratch.

Datasets

The authors evaluated PLADA across 14 diverse target datasets:

  • Reference Sets: ImageNet-1K (1.2M images) and ImageNet-21K (14.2M images).
  • Natural Images: Caltech-101, CIFAR-10, and Places365.
  • Fine-grained Tasks: CUB-200 (birds), Oxford-Flowers, Food-101, and RESISC45 (satellite imagery).
  • Medical (OOD) Tasks: BloodMNIST, DermaMNIST, and RetinaMNIST.

Key Insights and Ideas

  • The "Denoising" Effect: Surprisingly, training on a small, pruned subset (top 1–10% lowest energy) often yields higher accuracy than training on the full reference dataset. The pruning acts as a semantic filter, removing "distractor" images that don't align with the target task.
  • The Energy Paradox: For tasks very different from the reference set (like medical scans vs. ImageNet), the teacher's confidence is often misplaced. In these "far-OOD" scenarios, the authors found that selecting images with high energy (high uncertainty) actually worked better—a complete reversal of the standard strategy.
  • Labels are Information-Dense: A label is just a small integer. By associating these integers with high-resolution images already stored on-device, you get the "training signal" of a high-res dataset at a fraction of the communication cost.

Key Results

  • Extreme Compression: PLADA can transfer task knowledge with a payload of 85–206 KB (using a 1% keep rate on ImageNet-21K).
  • Performance: On the CUB-200 bird dataset, PLADA achieved high accuracy with a payload under 35 KB, while traditional data-centric baselines (like random sampling or coresets) failed to provide any viable training signal at that size.
  • Scale Matters: Using the larger ImageNet-21K as a reference set consistently outperformed ImageNet-1K, proving that a more diverse "background knowledge" base on the client leads to better task transfer.

Limitations and Future Work

  • Storage Overhead: The client must have the storage capacity to hold the large reference dataset (ImageNet-21K is ~1.3TB). However, the authors argue this is a one-time "sunk cost" that pays off as more tasks are served.
  • Training Time: Unlike receiving a pre-trained model, the client still has to spend compute cycles training the student model locally.
  • Beyond Classification: The current work is limited to classification. Extending this to regression or generative tasks (where labels aren't just simple integers) remains an open challenge.

By treating a shared dataset as a "codebook" for communication, PLADA offers a fascinating new path for deploying AI in the most disconnected corners of the world.

Summary updated

Abstract

A dataset server must often distribute the same large payload to many clients, incurring massive communication costs. Since clients frequently operate on diverse hardware and software frameworks, transmitting a pre-trained model is often infeasible; instead, agents require raw data to train their own task-specific models locally. While dataset distillation attempts to compress training signals, current methods struggle to scale to high-resolution data and rarely achieve sufficiently small files. In this paper, we propose Pseudo-Labels as Data (PLADA), a method that completely eliminates pixel transmission. We assume agents are preloaded with a large, generic, unlabeled reference dataset (e.g., ImageNet-1K, ImageNet-21K) and communicate a new task by transmitting only the class labels for specific images. To address the distribution mismatch between the reference and target datasets, we introduce a pruning mechanism that filters the reference dataset to retain only the labels of the most semantically relevant images for the target task. This selection process simultaneously maximizes training efficiency and minimizes transmission payload. Experiments on 10 diverse datasets demonstrate that our approach can transfer task knowledge with a payload of less than 1 MB while retaining high classification accuracy, offering a promising solution for efficient dataset serving.
BibTeX
@misc{shoshani2026dataset,
  author = {Elad Kimchi Shoshani and Leeyam Gabay and Yedid Hoshen},
  title = {{A Dataset is Worth 1 MB}},
  year = {2026},
  eprint = {2602.23358},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2602.23358},
}
Figure