TL;DR
This paper introduces LMTransplant, a novel text data augmentation technique that leverages Large Language Models (LLMs) to create more diverse and creative text variants. By "transplanting" original text into LLM-generated contexts and then regenerating it, LMTransplant significantly boosts downstream task performance, outperforming existing methods.
What Problem Does This Paper Address?
Deep learning models, especially in Natural Language Processing (NLP), are notoriously data-hungry. Insufficient training data can lead to poor generalization and overfitting. Data augmentation aims to address this by creating synthetic training samples from existing data. Traditional methods like back-translation or simple word-level transformations often produce variants that are too similar to the original text (lacking diversity) or disrupt the semantic coherence. While LLMs offer powerful "knowledge emergence" capabilities for augmentation, controlling their output for style and structure remains a challenge, often requiring complex prompt engineering. This paper tackles the problem of generating high-quality, diverse, and semantically coherent text augmentations that effectively leverage LLM capabilities without extensive manual prompt tuning.
Methods
The core of LMTransplant is the "transplant-then-regenerate" (TTR) strategy.
- Transplant: The original text is embedded into a context generated by an LLM. This is achieved through bidirectional text continuation. First, the LLM generates a subsequent context (left-to-right), and then it generates a preceding context (right-to-left) based on the combined original and subsequent text. This creates a richer, thematically aligned context for the original text.
- Regenerate: The original text is then masked within this expanded context. The LLM is prompted to regenerate a new variant that seamlessly fits into the context, preserves the original text's core attributes (theme, style, sentiment), but introduces novel content and creativity. This step is crucial for generating content-level variants rather than just rephrased ones.
The authors emphasize the use of prompt engineering to guide both the transplant and regeneration phases, ensuring the generated text is contextually relevant and semantically aligned.
Datasets
The authors evaluated LMTransplant on five established NLP benchmarks:
- SST-2: Sentiment analysis of movie reviews.
- TREC: Question classification.
- SNIPS: Human intent classification.
- MLQA: Cross-lingual question answering.
- CONLL-2003: Named Entity Recognition (NER).
For evaluation, they simulated low-resource scenarios by subsampling training data.
Key Insights and Ideas
- Beyond Rephrasing: LMTransplant moves beyond simple lexical or sentence-level rephrasing to generate content-level variants, injecting more creativity and diversity.
- Leveraging LLM Knowledge: The "transplant" phase effectively contextualizes the original text, allowing LLMs to tap into their vast embedded knowledge for more sophisticated augmentation.
- Bidirectional Context is Key: Using both preceding and subsequent context for regeneration significantly improves performance over unidirectional approaches, suggesting LLMs are more adept at generating context when they have a "forward" anchor.
- Exceptional Scalability: LMTransplant's performance continues to improve as more augmented samples are generated, unlike some other methods that plateau. This highlights its efficiency in utilizing LLM capabilities.
- Preserving Core Attributes: Despite generating novel content, the method is designed to maintain crucial aspects like theme, sentiment, and label consistency, making the augmented data practically useful.
Key Results
- Performance Gains: LMTransplant achieved significant performance improvements across all tested tasks. For example, it boosted accuracy on SST-2 by 28.16% and on MLQA by 23.66% compared to non-augmented data.
- Superior Quality: Intrinsic evaluations (Distinct-N for lexical diversity and Semantic Variability) showed LMTransplant's generated samples were more diverse and semantically richer than those from baselines like EDA, Back-translation, GPT3Mix, and AugGPT.
- Label Consistency: The method demonstrated strong label consistency, with augmented texts accurately retaining their original labels.
- Time Efficiency: While not the fastest (EDA is much quicker), LMTransplant is more efficient than methods like LLM2LLM, striking a balance between augmentation quality and computational cost.
Limitations and Future Work
- LLM Dependency: The effectiveness of LMTransplant is tied to the capabilities of the base LLM used. The current experiments used DeepSeek-V3, GPT-3.5-Turbo, and GPT-40. Future work will involve evaluating a wider range of LLMs.
- Task Adaptability: While effective for classification, QA, and NER, applying LMTransplant to other task types might require prompt adjustments. The authors plan to investigate its broader applicability.
- Prompt Engineering: Although the paper provides prompt templates, fine-tuning prompts for specific tasks or domains might still be necessary for optimal performance.
Summary updated