Disentangling Latent Shifts of In-Context Learning Through Self-Training

Josip Jukić, Jan SnajderSubmission to ICLR 2025 · Rejected

TL;DR

Standard in-context learning (ICL) is computationally expensive, vulnerable to demonstration ordering, and constrained by context windows. STICL (Self-Training ICL) resolves these issues by using a teacher LLM to generate pseudo-labels from demonstration-augmented prompts, distilling that context into a lightweight LoRA adapter for a student model. At test time, the student processes zero-shot queries alone, achieving better accuracy, substantially lower variance, and faster inference.

What Problem Does This Paper Address?

While Large Language Models (LLMs) excel at few-shot prompting, standard ICL suffers from three major flaws:

  1. Context length & efficiency bottlenecks: Concatenating multiple demonstrations alongside queries blows up inference latency (scaling quadratically with sequence length via self-attention).
  2. Extreme instability: LLM predictions fluctuate drastically depending on the specific selection and ordering of demonstrations.
  3. Imperfect latent disentanglement: Prior work has shown that ICL acts like an implicit gradient update (a "virtual weight update" or meta-gradient). However, previous attempts to extract or disentangle these shifts (e.g., In-Context Vectors or Batch-ICL) rely on linear-attention approximations or hidden-state heuristics, failing to capture the full non-linear complexity of modern transformer stacks.

Methods

STICL transfers the latent shifts induced by demonstrations directly into the model’s weights using a self-training, teacher-student paradigm:

[Demonstrations Xd + Query xq]  -->  Teacher LLM (Frozen Base)  -->  Pseudo-labels yt
                                                                             │ (Cross-Entropy Loss)
[Query xq only]                 -->  Student LLM (Base + LoRA)   -->  Predictions ys
  • Teacher Step: The frozen base model acts as the teacher, processing both the demonstrations ($X_d$) and unlabeled queries ($x_q$) to generate pseudo-labels.
  • Student Step: The student shares the exact same base model weights but attaches a small LoRA adapter (0.1–0.3% of total parameters). It receives only the query ($x_q$) as input and is fine-tuned to match the teacher's output via cross-entropy loss.
  • Shuffling for Stability (STICL-S): By shuffling demonstration order across training epochs, the adapter becomes robust to permutation order.
  • Adapter Arithmetic: To support large demonstration budgets without running out of context, demonstration sets can be partitioned into smaller chunks (e.g., subsets of 16), each trained into its own adapter, and then merged by simple parameter summation.

Datasets

The authors evaluated STICL using Llama 3 (8B), Phi 3 (mini 4k), and Llama 2 (7B) across standard benchmarks:

  • GLUE Benchmark: Evaluated on development sets across 7 tasks: binary single-sentence classification (CoLA, SST-2, RTE), sentence-pair tasks (MRPC, QQP, QNLI), and multi-class classification (MNLI).
  • MMLU: Evaluated on two sub-tasks with ample samples: elementary math (MATH) and miscellaneous (MISC).
  • Unlabeled Data: Fine-tuning used small pools of unlabeled queries ($|D_{\text{unlab}}| = 100$, scaling to 200 and 500 for GLUE).
  • No new benchmarks were introduced.

Key Insights and Ideas

  • Weak-to-Strong Generalization: Surprisingly, the student consistently outperformed its teacher (standard few-shot ICL). Rather than just memorizing the teacher's pseudo-labels, the student corrected erroneous ones over time (e.g., error correction rates increased steadily across epochs).
  • Local Consistency and Lower Lipschitz Constants: By computing the Frobenius norm of the input-output Jacobian matrix, the authors showed STICL achieves a significantly lower approximate Lipschitz constant than standard ICL and fine-tuning baselines, explaining why it is less sensitive to input perturbations.
  • Coverage Expansion: Prediction corrections were strongly correlated with proximity to confident examples (Pearson $r = -0.968$ for in-domain, $-0.916$ for out-of-domain). Confident predictions effectively ripple out to correct noisy neighborhood regions.
  • Demonstration Memorization vs. Generalization: Probing the trained adapters with the prompt "Repeat the demonstration word for word" yielded average BERTScore F1 similarities above $0.80$, proving the adapters genuinely internalized the demonstration text and latent task knowledge without seeing it directly during student forward passes.

Key Results

  • In-Domain (ID) Generalization (16-shot):
  • STICL-S outperformed standard 16-shot ICL across every GLUE and MMLU task, yielding absolute improvements of +2.6% to +11.9% on Llama 3 (e.g., RTE jumped from $75.1 \to 86.0$; MATH from $43.5 \to 49.5$) and +2.5% to +10.3% on Phi 3.
  • Consistently surpassed other disentanglement baselines like ICV and Batch-ICL, as well as Pattern-Based Fine-Tuning (PBFT) and MetaICL.
  • Stability Gains:
  • Testing over 50 varied prompt permutations revealed dramatic variance reduction (e.g., Llama 3 standard deviation on RTE dropped from 4.81 in standard ICL to 0.22 with STICL-S).
  • Out-of-Domain (OOD) Transfer:
  • When adapters were trained on one task and tested on a related unseen task (e.g., QNLI $\to$ RTE or QQP $\to$ MRPC), STICL-S consistently yielded higher scores and smaller standard deviations than ICL and ICV.
  • Adapter Scaling:
  • Merging up to 8 adapters (encoding a total of 128 demonstrations: $8 \times 16$) improved Llama 3 accuracy on RTE to 92.8% and MISC to 93.0%, far outperforming baselines.

Limitations and Future Work

  • Training Overhead & Break-Even Threshold: STICL requires backpropagation to train the adapter. For Llama 3 (8B) with 16 demonstrations and 100 unlabeled examples, the training compute breaks even with standard ICL inference after roughly 2,100 queries. Below this query volume, STICL is computationally more expensive upfront.
  • Need for Unlabeled Data: The approach assumes access to a modest amount of task-relevant, unlabeled queries (50–100 samples), which might be difficult to acquire in strictly data-scarce domains.
  • Massive Demonstration Sets: While scaling up to 128 demonstrations across 8 merged adapters proved successful, scaling to significantly larger demonstration pools or broader multi-task scenarios remains unexplored.

Summary updated

Abstract

In-context learning (ICL) has become essential in natural language processing, particularly with autoregressive large language models capable of learning from demonstrations provided within the prompt. However, ICL faces challenges with stability and long contexts, especially as the number of demonstrations grows, leading to poor generalization and inefficient inference. To address these issues, we introduce STICL (Self-Training ICL), an approach that disentangles the latent shifts of demonstrations from the latent shift of the query through self-training. STICL employs a teacher model to generate pseudo-labels and trains a student model using these labels, encoded in an adapter module. The student model exhibits weak-to-strong generalization, progressively refining its predictions over time. Our empirical results show that STICL improves generalization and stability, consistently outperforming traditional ICL methods and other disentangling strategies across both in-domain and out-of-domain data.
BibTeX
@misc{jukic2024disentangling,
  author = {Josip Jukić and Jan Snajder},
  title = {{Disentangling Latent Shifts of In-Context Learning Through Self-Training}},
  year = {2024},
  eprint = {2410.01508},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.01508},
}

Loading related content...

Figure