TL;DR
This paper investigates whether "smart" sample selection strategies for few-shot text augmentation with Large Language Models (LLMs) can outperform simple random selection. The findings suggest that for in-distribution tasks, random selection is often as good as, or even better than, more complex strategies, while for out-of-distribution tasks, a "synthetic samples dissimilarity" strategy shows promise but with caveats.
What Problem Does This Paper Address?
Large Language Models (LLMs) are increasingly used to augment text datasets, especially in low-resource settings, by generating new or paraphrased text samples. This is crucial for fine-tuning smaller downstream models. While LLM augmentation often uses "few-shot" prompting (providing examples to guide the LLM), the selection of these few-shot examples is typically done randomly. However, the broader field of few-shot learning has explored more sophisticated sample selection strategies (based on similarity, diversity, informativeness, etc.). This paper aims to bridge this gap by systematically investigating whether these informed strategies offer a significant advantage over random selection for LLM-based text augmentation, considering both in-distribution and out-of-distribution performance.
Methods
The authors conducted a comprehensive comparative study using 3 different LLMs (Llama-3.1, Mistral-v0.3, Gemma-2) and 8 diverse datasets across various NLP tasks. They compared 8 sample selection strategies against two baselines: zero-shot (no examples) and random few-shot. The augmentation techniques explored were paraphrasing existing samples and generating entirely new samples. The study evaluated downstream model performance (using RoBERTa-base) on both in-distribution and out-of-distribution test splits. The process was repeated multiple times with different random seeds to ensure robustness. The core of the methodology involved:
- Sample Selection Strategies: Investigating methods like Cosine Similarity/Dissimilarity, Forgetting (tracking model errors), Cartography (measuring learning difficulty), and Synthetic Samples Dissimilarity (generating synthetic data first).
- Augmentation Techniques: Paraphrasing existing text vs. generating entirely new text.
- Downstream Evaluation: Fine-tuning a RoBERTa-base model on augmented data and evaluating its performance on both original (in-distribution) and shifted (out-of-distribution) test sets.
- Low-Resource Setting: Using only 20 seed samples per label.
Datasets
The study utilized 8 datasets covering a range of NLP tasks:
- Sentiment Analysis: TweetEval, Yelp
- News Classification: AG News, News Topic
- Question Topic Classification: Yahoo, Trec
- Paraphrase Detection: QQP
- Natural Language Inference: MNLI
For out-of-distribution evaluation, they used test splits from datasets within the same domain but from different sources, or specifically designated OOD splits (e.g., PAWS for QQP).
Key Insights and Ideas
- Random is Often Good Enough (for ID): For in-distribution tasks, simple random sample selection for few-shot augmentation frequently performs as well as, or even better than, more complex, computationally expensive strategies. This suggests that for many standard augmentation needs, practitioners might not need to invest in sophisticated selection methods.
- OOD Shows Promise for Specific Strategies: The "Synthetic Samples Dissimilarity" strategy demonstrated a notable advantage for out-of-distribution performance, suggesting that generating diverse synthetic examples can indeed improve model robustness to domain shifts.
- No Silver Bullet: No single sample selection strategy consistently outperformed others across all datasets and conditions. The effectiveness of a strategy is highly dependent on the specific task and data distribution.
- Cost vs. Benefit: More advanced strategies often come with significant computational overhead (e.g., extra LLM inferences, partial model training). The marginal performance gains observed in many cases do not justify these increased costs.
- Paraphrasing vs. Generation: Paraphrasing was generally better for in-distribution performance, while generating new samples was more effective for out-of-distribution tasks.
Key Results
- In-Distribution (ID): The "Cartography (easy+ambiguous)" strategy was the best performing among the non-baseline strategies in 15.28% of cases. However, the random few-shot baseline performed best in 22.22% of cases, indicating its strong performance.
- Out-of-Distribution (OOD): The "Synthetic Samples Dissimilarity" strategy was the best performing in 31.94% of cases. The random few-shot baseline performed best in 11.11% of cases.
- Statistical Significance: While "Synthetic Samples Dissimilarity" showed promise for OOD, the performance increases were often not statistically significant.
- Overall Impact: Aggregated results showed that most sample selection strategies failed to provide a consistent, significant positive impact on model performance over the baselines, often showing small or even negative gains.
Limitations and Future Work
- Language and LLMs: The study was limited to English language datasets and specific LLM versions. Future work could explore multilingual settings and newer, larger LLMs.
- Prompt Engineering: The study used standard prompts. Investigating the impact of different prompt engineering techniques could be valuable.
- Downstream Fine-tuning: The study used PEFT methods for fine-tuning. Exploring other fine-tuning methods (like QLoRA) could be an extension.
- Dataset Bias: The study acknowledges the possibility that the LLMs might have been trained on some of the datasets used, potentially influencing results. Testing on entirely new, unpublished datasets would be beneficial.
- Feature Representation: Only one type of feature representation model was used for similarity-based strategies. Exploring others could yield different results.
- Computational Cost: The authors note that advanced strategies are expensive. Future work could focus on developing more cost-effective, yet effective, selection methods.
Summary updated