Feature Normalization and Cartography-Based Demonstrations for Prompt-Based Fine-Tuning on Emotion-Related Tasks

Mahshid Hosseini, Cornelia CarageaAAAI 2023 · DOI

Introduction

Large Language Models (LLMs) have demonstrated remarkable few-shot capabilities, yet their massive scale often precludes their use in specialized, resource-constrained environments. Smaller models like RoBERTa offer a more efficient alternative but frequently struggle when labeled data is scarce—a persistent challenge in niche domains such as empathy detection. Furthermore, few-shot models are notoriously sensitive to noise in both input text and labels.

This paper addresses these limitations by introducing a two-pronged strategy to enhance few-shot, prompt-based fine-tuning for emotion and empathy classification. The proposed approach combines "Moment Exchange" (MoEx) for feature-level data augmentation with "Cartography-Based Demonstrations" (CBDemo) for selecting stable, high-confidence examples.

Methods

The authors propose two novel techniques integrated into a prompt-based fine-tuning pipeline, as illustrated in the architecture below.

Architecture of the proposed framework featuring MoEx and CBDemo.

1. Moment Exchange (MoEx) Data Augmentation

Adapted from computer vision for NLP, MoEx treats the statistical "moments" (mean $\mu$ and standard deviation $\sigma$) of a hidden state representation as salient signals. During training, the model extracts these moments from one sample and swaps them with those of another sample within the same batch. This process forces the model to become more robust by learning to predict an interpolation of labels based on these blended features, effectively regularizing the model in low-data regimes.

2. Cartography-Based Demonstrations (CBDemo)

Standard prompt-based methods often rely on randomly selected examples as "demonstrations." However, random selection can include ambiguous or "hard-to-learn" samples that degrade performance. The authors utilize Dataset Cartography to categorize samples based on training dynamics:

  • Easy-to-learn: High confidence, low variability.
  • Ambiguous: High variability.
  • Hard-to-learn: Low confidence, low variability (often indicating mislabeled data).

CBDemo selects "easy-to-learn" samples that are semantically similar to the input (via SBERT embeddings) to serve as context. By providing the clearest, most reliable signals, CBDemo prevents the model from being misled by outliers.

Experimental Setup

The approach was evaluated on three emotion-related datasets:

  • MELD: Multi-party dialogues from the TV show Friends.
  • GoodNewsEveryone: News headlines with diverse emotion annotations.
  • iEmpathize: Online cancer forum posts categorized by empathy type.

Testing spanned three settings: MAIN (clean data), NOISYS (textual noise), and NOISYL (label noise using "hard-to-learn" samples as training data).

Key Results

The proposed method consistently outperformed baselines such as PET and GPT-3 in-context learning across all datasets and noise conditions.

Performance comparison on the MELD dataset across different shot counts and noise levels.

On the MELD dataset ($K=16$), the manual prompt-based fine-tuning with the proposed additions reached 47.72% accuracy, a significant jump from the 35.70% achieved by standard prompt-based fine-tuning. The method's robustness is particularly evident in the NOISYL (noisy label) setting for MELD ($K=8$), where it achieved 29.91% accuracy—nearly double the 13.94% of the standard baseline. Similarly, on the iEmpathize task ($K=4$), the method reached 67.80% accuracy compared to the 58.20% baseline.

Discussion and Conclusion

This research demonstrates that latent space augmentation and strategic demonstration selection are highly effective for improving small-model performance in few-shot scenarios. By swapping statistical moments in hidden layers, MoEx provides a powerful regularization tool for Transformers. Meanwhile, CBDemo highlights that in few-shot learning, the quality of demonstrations is more critical than quantity.

While the method introduces some computational overhead due to the initial dataset mapping phase, its ability to maintain high performance even under significant label noise makes it a robust choice for real-world applications where "clean" data is a luxury. Future work may explore scaling these techniques to larger architectures or more complex reasoning tasks.

Summary updated

BibTeX
@inproceedings{hosseini2023feature,
  author = {Mahshid Hosseini and Cornelia Caragea},
  title = {{Feature Normalization and Cartography-Based Demonstrations for Prompt-Based Fine-Tuning on Emotion-Related Tasks}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2023},
  doi = {10.1609/aaai.v37i11.26514},
}
Figure