TL;DR
This paper introduces DiLM, a novel method for text dataset distillation that generates synthetic text samples directly, overcoming limitations of previous embedding-based approaches. DiLM enables training models on these synthetic datasets that generalize better and are interpretable, making it a significant step towards more efficient and flexible text model development.
What Problem Does This Paper Address?
The success of large language models (LLMs) has led to a demand for massive datasets and significant computational resources for training. Dataset distillation aims to compress training datasets into smaller, more informative synthetic datasets, reducing training costs. However, existing text dataset distillation methods primarily operate at the word embedding level. This approach creates synthetic datasets that are unreadable to humans and, crucially, cannot be used to train models with different word embedding weights, limiting their practical applicability. The authors address this gap by proposing a method that distills datasets into human-readable text, enhancing model-agnostic applicability and interpretability.
Methods
DiLM tackles the challenge of discrete text by employing a language model as a surrogate for gradient-based optimization. The process involves three main steps:
- Synthetic Training Data Generation: A language model (e.g., GPT-2) is pre-trained to generate synthetic text samples that mimic the distribution of the original dataset. For classification tasks, class-specific tokens are used to guide generation.
- Fine-tuning for Gradient Matching: The pre-trained generator model is fine-tuned to produce "informative" synthetic samples. This is achieved by minimizing a gradient matching loss, which aligns the gradients of a learner model trained on synthetic data with those trained on real data. To overcome the non-differentiability of discrete text, DiLM uses a differentiable backward pass by weighting the learner's loss with the generation probabilities of the synthetic samples. Two techniques are introduced to improve this process:
- Representative Teacher: K-centers clustering is used to select representative samples from the original dataset to act as a more robust "teacher" for gradient matching.
- Diverse Mini-batch Sampling: To ensure the generator explores a diverse range of synthetic samples, generated samples are clustered, and one sample is randomly chosen from each cluster for each training step.
- Generate Synthetic Dataset: The fine-tuned generator model is used to produce a large number of synthetic samples, from which representative samples are selected using K-centers clustering to form the final distilled dataset.
Datasets
The authors evaluated DiLM on three text classification datasets from the GLUE benchmark:
- SST-2: A binary sentiment classification task for movie reviews.
- QQP: A task to determine if two questions are semantically equivalent.
- MNLI-m: A natural language inference task.
Key Insights and Ideas
- Text-level distillation is crucial for applicability: Unlike embedding-level distillation, DiLM's text-level output is interpretable and can be used to train models with different architectures and embedding weights, a significant advantage for practical applications.
- Language models as surrogates for discrete optimization: Using a language model to generate synthetic text bypasses the direct optimization of discrete tokens, enabling gradient-based methods for text dataset distillation.
- Gradient matching is effective for text: The core idea of matching gradients between models trained on real and synthetic data proves effective even for discrete text, leading to informative synthetic samples.
- Representative teachers and diverse sampling improve quality: Employing K-centers for selecting representative real samples and diverse mini-batch sampling for synthetic samples significantly enhances the quality and effectiveness of the distilled datasets.
- DiLM generalizes well: The distilled datasets demonstrate strong generalization capabilities, not only for training the same model used for distillation but also for training entirely different models and for few-shot in-context learning with LLMs.
Key Results
- DiLM-distilled datasets consistently outperformed coreset selection methods (Random, K-centers, Herding) on SST-2, QQP, and MNLI-m when training BERTBASE.
- DiLM achieved remarkable cross-model generalization, performing well when training models like RoBERTaBASE, BERTLARGE, and XLNetBASE, which differ in architecture or pre-training objectives from the source BERTBASE model.
- For in-context learning of LLMs (GPT-2-XL, OPT, Llama 2), DiLM's distilled datasets also showed superior performance compared to Random and K-centers baselines.
- An ablation study confirmed that all three proposed techniques (representative teacher, diverse mini-batch sampling, and K-centers selection) contribute positively to DiLM's performance.
- The performance of DiLM generally scaled with the size of the distilled synthetic dataset (Data Per Class - DPC).
Limitations and Future Work
The authors acknowledge several limitations:
- Performance Gap: While DiLM shows significant improvement, there is still a performance gap compared to training on the full dataset. They suggest using larger language models for generation or exploring other distillation objectives to further improve performance.
- Task Specificity: The current evaluation is limited to text classification tasks. Future work could explore DiLM for text generation tasks and full-scratch training of language models.
- Privacy Preservation: While dataset distillation can be applied to privacy preservation, DiLM's potential for memorizing real samples makes it less suitable for this specific application. However, they argue that the benefits of text-level distillation for model-agnostic training outweigh this limitation for general practical applications.
Summary updated