In-context Learning Demonstration Generation with Text Distillation

Wuyuqing Wang, Erkun Yang, Zilan Zhou, Cheng DengIJCAI 2025 · DOI

Abstract

In-Context Learning (ICL) allows Large Language Models (LLMs) to perform tasks using a few demonstrations. However, performance is highly sensitive to the specific examples chosen. This paper introduces Distillation-based Demonstration Generation (DDG), a framework that moves beyond simple example selection. DDG synthesizes "super-demonstrations" that encapsulate the essential patterns of an entire dataset through gradient matching. Experimental results across ten benchmarks show that DDG significantly improves accuracy and stability compared to traditional selection methods.

Introduction: The Brittleness of Selection

In-Context Learning (ICL) is a powerful paradigm, but it is notoriously "brittle." Minor variations in the choice or ordering of demonstrations can lead to drastic swings in model accuracy. Existing solutions typically rely on demonstration selection, which picks the "best" samples from a dataset. This approach suffers from two main drawbacks:

  1. Information Loss: By selecting only a few samples, the vast majority of the training data—and the patterns within it—are discarded.
  2. Lack of Generalization: Examples optimized for one LLM often fail to perform as well when transferred to another.

The core challenge addressed here is how to condense the knowledge of thousands of data points into a handful of highly representative, synthetic demonstrations.

Methodology: The DDG Framework

The proposed DDG framework shifts the paradigm from selection to synthesis. It employs a generative model to create new demonstrations that are optimized to represent the entire dataset's distribution.

The architecture of the DDG framework, illustrating the interaction between the generative model, calculative models, and the gradient matching process.

The architecture consists of three primary components:

  • Generative Model ($G_\phi$): A GPT-3 based model that synthesizes the textual demonstrations.
  • Calculative Models ($U_\theta$): Smaller models (e.g., RoBERTa-large) that evaluate the representativeness of the generated text.
  • Gradient Matching: This is the optimization objective. The generative model is trained to minimize the discrepancy between the gradients of a model trained on the original dataset and one trained on the synthetic samples. This ensures that the synthetic data "moves" the model parameters in the same direction as the real data.

To stabilize this process for discrete text, the authors implement a Teacher-Student framework using Exponential Moving Average (EMA) fitting. This prevents the generator from producing repetitive or low-quality text, ensuring diverse and informative output.

Experimental Evaluation

The researchers evaluated DDG on ten text classification benchmarks, ranging from short-text sentiment analysis (SST-2, CR) to long-text multi-tag intent detection (BANKING77) and fine-grained sentiment (GoEmotions).

Key Results and Insights

DDG consistently outperformed state-of-the-art selection and distillation baselines across multiple LLMs, including LLaMA-2, Qwen, and Mistral.

  • Performance Gains: DDG improved classification accuracy by an average of 7% on short-text tasks and 5% on long-text tasks like BANKING77.
  • Information Density: The framework effectively performs "textual compression." For instance, on the GoEmotions dataset, synthetic demonstrations achieved performance parity with the original data while being 70-75% shorter.
  • The Overwriting Threshold: LLMs exhibit a "critical threshold" for prompt length. While more demonstrations generally help, performance eventually degrades as the model loses focus. DDG maximizes information density, providing more value before this threshold is reached.
Classification accuracy on the BANKING77 dataset compared against the LongICLBench baseline as token length increases.

Conclusion and Future Work

DDG represents a significant step forward in making ICL more robust and efficient. By synthesizing demonstrations rather than selecting them, the framework captures universal task features that generalize well across different LLMs.

However, the approach has limitations. The nested-loop optimization for gradient matching is computationally more expensive than simple similarity-based selection. Furthermore, the quality of synthesis depends on the calculative model's ability to learn the underlying task. Future research will explore extending DDG to non-classification tasks, such as mathematical reasoning and summarization.

Summary updated

BibTeX
@inproceedings{wang2025context,
  author = {Wuyuqing Wang and Erkun Yang and Zilan Zhou and Cheng Deng},
  title = {{In-context Learning Demonstration Generation with Text Distillation}},
  booktitle = {International Joint Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.24963/ijcai.2025/716},
}
Figure