TL;DR
This paper introduces Mini-DA, a novel method for data augmentation using Large Language Models (LLMs) that significantly reduces computational costs. By intelligently selecting only the most challenging samples for augmentation based on model feedback, Mini-DA achieves performance comparable to full augmentation with a fraction of the data and computational resources.
What Problem Does This Paper Address?
Deep learning models thrive on data, but acquiring high-quality, labeled data is often expensive and time-consuming. Data augmentation techniques help by artificially expanding datasets, but current LLM-based augmentation methods, while powerful, demand substantial computational resources for generating vast amounts of synthetic data. This creates a bottleneck, especially for resource-constrained projects or when aiming for state-of-the-art performance. The paper addresses this by proposing a more efficient way to leverage LLMs for data augmentation, focusing on maximizing impact while minimizing cost.
Methods
The core of Mini-DA is an iterative, LLM-in-the-loop framework. Here's how it works:
- Dataset Splitting: The original dataset is split into training, validation, and a held-out test set. The training and validation sets are then further divided into k folds for cross-validation.
- Model Training: A target model (e.g., a BERT-based model for text classification) is trained on k-1 folds and validated on the remaining fold, repeating this k times.
- Validation Set Prediction: The trained models are used to predict on their respective validation sets.
- Challenging Case Collection: Samples where the model makes incorrect predictions on the validation set are identified as "challenging cases."
- Selective Data Augmentation: These challenging cases are then fed to an instruction-tuned LLM (GPT-3.5 Turbo in this study) with specific prompts to generate synthetic, augmented data.
- Augmented Dataset Integration: The newly generated data is added to an augmented dataset. In subsequent epochs, when the training/validation folds are re-split, any augmented data corresponding to samples in the new training folds is integrated.
- Iterative Process: Steps 2-6 are repeated for a set number of epochs or until the model's performance on the test set plateaus, indicating convergence.
The key innovation is the selective augmentation based on model feedback, rather than a blanket augmentation of the entire dataset.
Datasets
The authors evaluated their approach on two intent detection datasets:
- banking77-filtered: A subset of the original banking77 dataset, focusing on the banking domain with 77 intent labels. The filtered version used for experiments had 2,047 training and 693 test cases.
- ECDT-NLU-2019: A Chinese natural language understanding dataset, where only the intent detection task was considered. This dataset comprised 2,061 training and 516 test cases with 45 intent labels.
Key Insights and Ideas
- Feedback is Key: The most significant insight is that not all data points are equally beneficial for augmentation. Focusing LLM generation efforts on samples the model struggles with yields better results than augmenting everything. This is analogous to how human learners focus on difficult concepts.
- Efficiency Gains: By being selective, Mini-DA drastically reduces the number of LLM calls and the amount of generated data needed. This translates directly to lower computational costs and faster iteration cycles.
- Comparable Performance: The paper demonstrates that this targeted approach can achieve performance levels very close to, and sometimes even exceeding, those obtained with full data augmentation. This suggests that the "quality" of augmented data, not just the quantity, is paramount.
- Iterative Refinement: The iterative nature of the framework allows the model to continuously identify new challenging samples as it improves, creating a virtuous cycle of learning and augmentation.
Key Results
- banking77-filtered: Mini-DA achieved an average accuracy of 86.78% on the test set, which was 0.37% higher than the fully augmented dataset's performance (86.41%). Crucially, this was achieved by augmenting only 28% of the training data at its optimal epoch.
- ECDT-NLU-2019: On this dataset, Mini-DA achieved an average accuracy of 92.02% at the fourth augmentation epoch, only 0.11% lower than the fully augmented dataset (92.13%). This was accomplished by augmenting just 13% of the training data, representing a significant reduction in LLM requests.
- Cost Reduction: The paper highlights that Mini-DA significantly reduces computational costs by minimizing LLM usage. For instance, on ECDT-NLU-2019, it reduced GPT-3.5 Turbo requests by 1793 compared to full augmentation.
Limitations and Future Work
- Limitations: The paper acknowledges that the current framework is primarily demonstrated for intent detection tasks. While the core idea is generalizable, its effectiveness on other NLP tasks would require further validation. The computational cost of the iterative training and prediction steps, though less than full augmentation, is still a factor.
- Future Work: The authors plan to conduct comprehensive experiments across a wider range of NLP tasks, including question answering, text generation, and text retrieval. They believe Mini-DA can be effective in improving model performance across these diverse tasks in a data-efficient manner.
Summary updated