TL;DR
This paper explores the feasibility of applying dataset distillation—a technique that compresses massive datasets into a few highly informative synthetic images—to medical imaging. The authors demonstrate that while current distillation methods excel on medical datasets with large, distinct variations between classes, they struggle with datasets characterized by subtle, fine-grained diagnostic differences. Crucially, they discover that the performance of a simple random data selection baseline can serve as a reliable heuristic to predict how successful dataset distillation will be.
What Problem Does This Paper Address?
Sharing medical image datasets across institutions is vital for training robust machine learning models, but it is heavily restricted by patient privacy regulations and the sheer logistical challenge of transferring massive files. Dataset distillation (DD) offers a clever workaround: instead of sharing the entire dataset, you synthesize a tiny fraction of "distilled" images that retain the core optimization gradient or training trajectory of the original data.
While DD has achieved remarkable success on natural image benchmarks like CIFAR and ImageNet, its applicability to medical imaging has remained an open question. Natural images typically feature large inter-class variations (e.g., distinguishing a chair from a bicycle). In contrast, medical images often require expert knowledge to distinguish between highly similar biological tissues, where the difference between a benign and malignant tumor might hinge on subtle textures or minute abnormalities.
Methods
The authors establish a universal medical dataset distillation pipeline to evaluate two state-of-the-art distillation paradigms:
- Dataset Condensation (DC): This method synthesizes a small dataset by matching the network gradients calculated from the synthetic data to those calculated from the original, full dataset.
- Matching Training Trajectories (MTT): This approach trains a "student" network on the synthetic data and forces its parameter trajectory to align with the training trajectory of a "teacher" network trained on the real dataset.
To rigorously test these methods, the authors designed two types of experiments:
- Simulation of Large Variation: They created an integrated dataset by taking 10% of eight different medical imaging modalities and relabeling them by their modality name rather than their clinical diagnosis. This simulated the large inter-class variation typical of natural image datasets.
- Individual Dataset Distillation: They applied DC and MTT directly to individual medical datasets at varying densities of Images Per Class (IPC = 1, 10, and 50) to see how they handle realistic clinical classification tasks.
Datasets
The study evaluates the distillation methods using eight diverse, preprocessed datasets from the MedMNIST collection, spanning six different modalities and ranging in scale from 10,000 to over 165,000 training images:
- PathMNIST (Colon pathology)
- DermaMNIST (Dermatoscopy)
- OCTMNIST (Retinal OCT)
- BloodMNIST (Blood cell microscopy)
- TissueMNIST (Kidney cortex microscopy)
- OrganAMNIST, OrganCMNIST, OrganSMNIST (Abdominal CT scans)
All images were standardized to a uniform $32 \times 32$ pixel dimension and preprocessed using ZCA whitening to reduce redundancy.
Key Insights and Ideas
- The Inter-Class Variation Bottleneck: Dataset distillation is not a one-size-fits-all solution for medical data. Standard DD algorithms excel when classes look visually distinct (e.g., BloodMNIST cells), but they break down when differences are subtle (e.g., PathMNIST tissue types).
- Random Selection Can Outperform Distillation: Surprisingly, for datasets with low inter-class variation (like PathMNIST and DermaMNIST), simply selecting 50 random real images per class yielded better downstream model accuracy than using 50 distilled synthetic images.
- A Predictive Heuristic for DD Success: The authors discovered a strong linear correlation between the accuracy achieved by a randomly selected subset and the accuracy achieved by a distilled dataset. This means researchers can use a quick, computationally cheap random selection test as a proxy to predict whether a dataset will benefit from distillation.
Key Results
- High-Variation Success: In the simulated multi-modality dataset, the DC method achieved over 90% accuracy with just 10 images per class, vastly outperforming the random selection baseline.
- BloodMNIST: MTT achieved an impressive 89.18% accuracy at IPC = 50, closing in on the full dataset accuracy of 95.80%.
- The Failure Modes: On PathMNIST, random selection achieved 73.47% accuracy at IPC = 50, whereas MTT only reached 68.61% and DC collapsed to 38.26%. This highlights that current distillation methods struggle to synthesize the fine-grained features necessary for complex pathology classification.
Limitations and Future Work
The primary limitation identified is that existing dataset distillation algorithms—originally engineered for natural images—fail to capture the nuanced, fine-grained diagnostic details critical to many medical imaging tasks. When the visual differences between classes are minute, the distillation process tends to lose essential medical information.
The authors conclude that future work must move away from generic computer vision distillation methods and focus on developing specialized medical dataset distillation algorithms that explicitly prioritize the preservation of subtle textures, densities, and localized anomalies.
Summary updated