DemoShapley: Valuation of Demonstrations for In-Context Learning

Shan Xie, Man Luo, Chadly Daniel Stern, Mengnan Du, Lu Cheng

Abstract

Large language models (LLMs) using in-context learning (ICL) excel in many tasks without task-specific fine-tuning. However, demonstration selection and ordering greatly impact ICL effectiveness. Focus on this issue, we propose DemoShapley, a Shapley-value based method that evaluates each demonstration's contribution by measuring its marginal effect across different prompt permutations. To further account for ICL's limited context windows and frequent low-shot settings, we introduce Beta-DemoShapley, a weighted extension that emphasizes the influence of smaller prompt sizes. Experiments on multiple benchmarks show that DemoShapley consistently outperforms existing influence-based selection strategies, while Beta-DemoShapley further improves performance in low-shot scenarios. Both methods also detect mislabeled data, enhance generalization to out-of-distribution tasks, and reduce demographic bias. Together, they provide a unified and robust framework for demonstration valuation in ICL.
BibTeX
@misc{xie2024demoshapley,
  author = {Shan Xie and Man Luo and Chadly Daniel Stern and Mengnan Du and Lu Cheng},
  title = {{DemoShapley: Valuation of Demonstrations for In-Context Learning}},
  year = {2024},
  eprint = {2410.07523},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.07523},
}

Loading related content...

Figure