Introduction
Large Language Models (LLMs) have demonstrated remarkable few-shot capabilities, yet their massive scale often precludes their use in specialized, resource-constrained environments. Smaller models like RoBERTa offer a more efficient alternative but frequently struggle when labeled data is scarce—a persistent challenge in niche domains such as empathy detection. Furthermore, few-shot models are notoriously sensitive to noise in both input text and labels.
This paper addresses these limitations by introducing a two-pronged strategy to enhance few-shot, prompt-based fine-tuning for emotion and empathy classification. The proposed approach combines "Moment Exchange" (MoEx) for feature-level data augmentation with "Cartography-Based Demonstrations" (CBDemo) for selecting stable, high-confidence examples.
Methods
The authors propose two novel techniques integrated into a prompt-based fine-tuning pipeline, as illustrated in the architecture below.
1. Moment Exchange (MoEx) Data Augmentation
Adapted from computer vision for NLP, MoEx treats the statistical "moments" (mean $\mu$ and standard deviation $\sigma$) of a hidden state representation as salient signals. During training, the model extracts these moments from one sample and swaps them with those of another sample within the same batch. This process forces the model to become more robust by learning to predict an interpolation of labels based on these blended features, effectively regularizing the model in low-data regimes.
2. Cartography-Based Demonstrations (CBDemo)
Standard prompt-based methods often rely on randomly selected examples as "demonstrations." However, random selection can include ambiguous or "hard-to-learn" samples that degrade performance. The authors utilize Dataset Cartography to categorize samples based on training dynamics:
- Easy-to-learn: High confidence, low variability.
- Ambiguous: High variability.
- Hard-to-learn: Low confidence, low variability (often indicating mislabeled data).
CBDemo selects "easy-to-learn" samples that are semantically similar to the input (via SBERT embeddings) to serve as context. By providing the clearest, most reliable signals, CBDemo prevents the model from being misled by outliers.
Experimental Setup
The approach was evaluated on three emotion-related datasets:
- MELD: Multi-party dialogues from the TV show Friends.
- GoodNewsEveryone: News headlines with diverse emotion annotations.
- iEmpathize: Online cancer forum posts categorized by empathy type.
Testing spanned three settings: MAIN (clean data), NOISYS (textual noise), and NOISYL (label noise using "hard-to-learn" samples as training data).
Key Results
The proposed method consistently outperformed baselines such as PET and GPT-3 in-context learning across all datasets and noise conditions.
On the MELD dataset ($K=16$), the manual prompt-based fine-tuning with the proposed additions reached 47.72% accuracy, a significant jump from the 35.70% achieved by standard prompt-based fine-tuning. The method's robustness is particularly evident in the NOISYL (noisy label) setting for MELD ($K=8$), where it achieved 29.91% accuracy—nearly double the 13.94% of the standard baseline. Similarly, on the iEmpathize task ($K=4$), the method reached 67.80% accuracy compared to the 58.20% baseline.
Discussion and Conclusion
This research demonstrates that latent space augmentation and strategic demonstration selection are highly effective for improving small-model performance in few-shot scenarios. By swapping statistical moments in hidden layers, MoEx provides a powerful regularization tool for Transformers. Meanwhile, CBDemo highlights that in few-shot learning, the quality of demonstrations is more critical than quantity.
While the method introduces some computational overhead due to the initial dataset mapping phase, its ability to maintain high performance even under significant label noise makes it a robust choice for real-world applications where "clean" data is a luxury. Future work may explore scaling these techniques to larger architectures or more complex reasoning tasks.
Summary updated