Dataset Distillation in Medical Imaging: A Feasibility Study

Muyang Li, Can Cui, Quan Liu, Ruining Deng, Tianyuan Yao, Marilyn Lionts, Yuankai Huo

TL;DR

This paper explores the feasibility of applying dataset distillation—a technique that compresses massive datasets into a few highly informative synthetic images—to medical imaging. The authors demonstrate that while current distillation methods excel on medical datasets with large, distinct variations between classes, they struggle with datasets characterized by subtle, fine-grained diagnostic differences. Crucially, they discover that the performance of a simple random data selection baseline can serve as a reliable heuristic to predict how successful dataset distillation will be.

What Problem Does This Paper Address?

Sharing medical image datasets across institutions is vital for training robust machine learning models, but it is heavily restricted by patient privacy regulations and the sheer logistical challenge of transferring massive files. Dataset distillation (DD) offers a clever workaround: instead of sharing the entire dataset, you synthesize a tiny fraction of "distilled" images that retain the core optimization gradient or training trajectory of the original data.

While DD has achieved remarkable success on natural image benchmarks like CIFAR and ImageNet, its applicability to medical imaging has remained an open question. Natural images typically feature large inter-class variations (e.g., distinguishing a chair from a bicycle). In contrast, medical images often require expert knowledge to distinguish between highly similar biological tissues, where the difference between a benign and malignant tumor might hinge on subtle textures or minute abnormalities.

Figure 1: Difference between natural and medical datasets

Methods

The authors establish a universal medical dataset distillation pipeline to evaluate two state-of-the-art distillation paradigms:

  1. Dataset Condensation (DC): This method synthesizes a small dataset by matching the network gradients calculated from the synthetic data to those calculated from the original, full dataset.
  2. Matching Training Trajectories (MTT): This approach trains a "student" network on the synthetic data and forces its parameter trajectory to align with the training trajectory of a "teacher" network trained on the real dataset.

To rigorously test these methods, the authors designed two types of experiments:

  • Simulation of Large Variation: They created an integrated dataset by taking 10% of eight different medical imaging modalities and relabeling them by their modality name rather than their clinical diagnosis. This simulated the large inter-class variation typical of natural image datasets.
  • Individual Dataset Distillation: They applied DC and MTT directly to individual medical datasets at varying densities of Images Per Class (IPC = 1, 10, and 50) to see how they handle realistic clinical classification tasks.
Figure 2: General working pipeline for this paper. We investigate the effectiveness of distillation on medical datasets by designing multiple experiments to answer these questions.

Datasets

The study evaluates the distillation methods using eight diverse, preprocessed datasets from the MedMNIST collection, spanning six different modalities and ranging in scale from 10,000 to over 165,000 training images:

  • PathMNIST (Colon pathology)
  • DermaMNIST (Dermatoscopy)
  • OCTMNIST (Retinal OCT)
  • BloodMNIST (Blood cell microscopy)
  • TissueMNIST (Kidney cortex microscopy)
  • OrganAMNIST, OrganCMNIST, OrganSMNIST (Abdominal CT scans)

All images were standardized to a uniform $32 \times 32$ pixel dimension and preprocessed using ZCA whitening to reduce redundancy.

Key Insights and Ideas

  • The Inter-Class Variation Bottleneck: Dataset distillation is not a one-size-fits-all solution for medical data. Standard DD algorithms excel when classes look visually distinct (e.g., BloodMNIST cells), but they break down when differences are subtle (e.g., PathMNIST tissue types).
  • Random Selection Can Outperform Distillation: Surprisingly, for datasets with low inter-class variation (like PathMNIST and DermaMNIST), simply selecting 50 random real images per class yielded better downstream model accuracy than using 50 distilled synthetic images.
  • A Predictive Heuristic for DD Success: The authors discovered a strong linear correlation between the accuracy achieved by a randomly selected subset and the accuracy achieved by a distilled dataset. This means researchers can use a quick, computationally cheap random selection test as a proxy to predict whether a dataset will benefit from distillation.

Key Results

  • High-Variation Success: In the simulated multi-modality dataset, the DC method achieved over 90% accuracy with just 10 images per class, vastly outperforming the random selection baseline.
  • BloodMNIST: MTT achieved an impressive 89.18% accuracy at IPC = 50, closing in on the full dataset accuracy of 95.80%.
  • The Failure Modes: On PathMNIST, random selection achieved 73.47% accuracy at IPC = 50, whereas MTT only reached 68.61% and DC collapsed to 38.26%. This highlights that current distillation methods struggle to synthesize the fine-grained features necessary for complex pathology classification.
Figure 3: Distillation simulation on a large inter-class variance integrated medical dataset. The original datasets, from left to right, are TissueMNIST, PathMNIST, OrganSMNIST, OrganCMNIST, OrganAMNIST, OCTM- NIST, DermaMNIST, and BloodMNIST, each treated as a single class in this integrated dataset. The right figure shows that higher distillation accuracy highlights the potential for increasing inter-class variance in spe- cific medical datasets for improved distillation.

Limitations and Future Work

The primary limitation identified is that existing dataset distillation algorithms—originally engineered for natural images—fail to capture the nuanced, fine-grained diagnostic details critical to many medical imaging tasks. When the visual differences between classes are minute, the distillation process tends to lose essential medical information.

The authors conclude that future work must move away from generic computer vision distillation methods and focus on developing specialized medical dataset distillation algorithms that explicitly prioritize the preservation of subtle textures, densities, and localized anomalies.

Summary updated

Abstract

Data sharing in the medical image analysis field has potential yet remains underappreciated. The aim is often to share datasets efficiently with other sites to train models effectively. One possible solution is to avoid transferring the entire dataset while still achieving similar model performance. Recent progress in data distillation within computer science offers promising prospects for sharing medical data efficiently without significantly compromising model effectiveness. However, it remains uncertain whether these methods would be applicable to medical imaging, since medical and natural images are distinct fields. Moreover, it is intriguing to consider what level of performance could be achieved with these methods. To answer these questions, we conduct investigations on a variety of leading data distillation methods, in different contexts of medical imaging. We evaluate the feasibility of these methods with extensive experiments in two aspects: 1) Assess the impact of data distillation across multiple datasets characterized by minor or great variations. 2) Explore the indicator to predict the distillation performance. Our extensive experiments across multiple medical datasets reveal that data distillation can significantly reduce dataset size while maintaining comparable model performance to that achieved with the full dataset, suggesting that a small, representative sample of images can serve as a reliable indicator of distillation success. This study demonstrates that data distillation is a viable method for efficient and secure medical data sharing, with the potential to facilitate enhanced collaborative research and clinical applications.
BibTeX
@misc{li2024dataset,
  author = {Muyang Li and Can Cui and Quan Liu and Ruining Deng and Tianyuan Yao and Marilyn Lionts and Yuankai Huo},
  title = {{Dataset Distillation in Medical Imaging: A Feasibility Study}},
  year = {2024},
  eprint = {2407.14429},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2407.14429},
}
Figure