Disentangling Latent Shifts of In-Context Learning with Weak Supervision

Josip Jukić, Jan Šnajder

TL;DR

In-Context Learning (ICL) allows large language models to learn from a few examples in a prompt, but it suffers from high inference costs and extreme sensitivity to example ordering. This paper introduces WILDA, a method that treats ICL as a source of weak supervision to train a lightweight LoRA adapter on unlabeled queries, effectively "baking" the demonstrations directly into the model's weights. Surprisingly, the resulting student model not only eliminates the need for long prompts but actually outperforms its ICL teacher through weak-to-strong generalization.

What Problem Does This Paper Address?

Large Language Models (LLMs) are famous for their ability to perform few-shot learning via In-Context Learning (ICL)—simply prepend a few labeled examples (demonstrations) to a query, and the model figures out the task without any weight updates.

However, ICL has two major flaws:

  1. Instability: LLMs are notoriously fickle. Changing the choice or even just the ordering of the demonstrations can cause accuracy to swing wildly.
  2. Inefficiency: Prepending examples forces the model to process long contexts for every single query. Because transformer attention scales quadratically with sequence length, this inflates compute costs, introduces positional biases, and eats up the model's context window.

Prior attempts to solve this tried to "disentangle" the effect of demonstrations from the query by hacking into the model's internal attention heads or hidden states. However, these methods rely on simplified linear approximations of attention that fail to capture the complex, non-linear dynamics of real-world transformers.

Methods

The authors propose WILDA (Weakly-supervised In-context Learning Disentanglement via Adapters). Instead of manipulating internal mechanics, WILDA takes a functional approach using a teacher-student framework:

Illustration of the WILDA teacher-student framework.
  1. The Teacher: A standard LLM is prompted with the few-shot demonstrations alongside an unlabeled query to generate a "pseudo-label" (the full probability distribution over the vocabulary).
  2. The Student: The exact same LLM, but equipped with a lightweight LoRA adapter, is given only the query (no demonstrations). It is trained to match the teacher's output distribution by minimizing cross-entropy loss.

By doing this, the adapter internalizes the latent shift induced by the demonstrations. The authors experiment with three training variants, finding that shuffling the demonstration order between training epochs (WILDA-S) yields the highest stability. Furthermore, because the knowledge is stored in modular adapters, multiple adapters trained on different demonstration subsets can be combined using simple adapter arithmetic (adding their weights together).

Datasets

The method was evaluated using Llama 3 (8B), Phi 3 (mini 4k), and Llama 2 (7B) across several standard benchmarks:

  • GLUE Benchmark: 7 tasks covering single-sequence and sequence-pair classification (RTE, SST, QNLI, MNLI, COLA, MRPC, QQP).
  • MMLU Benchmark: Selected subsets including "elementary math" (MATH) and "miscellaneous" (MISC).
  • ARC-Challenge: Used to evaluate more demanding multi-hop reasoning.

For training, WILDA only required a small pool of 100 to 500 unlabeled instances per task to act as queries for the teacher.

Key Insights and Ideas

  • Weak-to-Strong Generalization: The most fascinating finding is that the student regularly surpasses the teacher. Even though the teacher's pseudo-labels are noisy and inconsistent, the student filters out this noise through pseudo-label correction (enforcing local consistency in the representation space) and coverage expansion (generalizing beyond the narrow scope of the few demonstrations).
  • Smoother Loss Landscapes: Empirical analysis showed that WILDA significantly lowers the model's approximated Lipschitz constant compared to standard ICL. In plain English: the model becomes much less sensitive to small perturbations in the input, explaining its massive boost in stability.
  • True Disentanglement: By testing the student's ability to reconstruct the original demonstrations word-for-word from the adapter weights alone, the authors proved that the adapters faithfully and cleanly encode the specific task context.

Key Results

  • Superior Accuracy: On Llama 3 (8B), WILDA-S outperformed standard ICL across all datasets, yielding absolute accuracy improvements between 2.6% and 11.9%.
Table 1: ID generalization scores for the 16-shot setup and |Dunlab| = 100. The standard deviations of 10 runs are shown as subscripts. The highest scores and smallest standard deviations are highlighted in bold, while the second-best scores are underlined.
  • Unmatched Stability: When shuffling the order of 16 demonstrations 50 times, standard ICL showed high variance (e.g., a standard deviation of 4.81 on the RTE dataset). WILDA-S crushed this instability, bringing the standard deviation down to 0.22.
Table 2: Standard deviations of generalization scores across 50 runs with varied orderings of 16 demonstrations. The smallest deviations are in bold, and the second-smallest are underlined.
  • Amortized Efficiency: While training an adapter incurs an upfront cost, removing demonstrations speeds up inference drastically. The authors calculated that for a 16-shot setup on Llama 3, the computational cost of fine-tuning breaks even after roughly 2,100 inferences, after which every query is pure profit in terms of speed and compute savings.

Limitations and Future Work

  • Data & Compute Overhead: WILDA is not a zero-cost solution. It requires a small set of unlabeled data and an initial fine-tuning phase (10 epochs), making it less ideal for scenarios where you only need to process a handful of queries.
  • Massive Demonstration Scales: While the authors successfully fused up to 8 adapters (encoding 128 total demonstrations), the behavior and scalability of WILDA when dealing with thousands of demonstrations remains unexplored.

Summary updated

Abstract

In-context learning (ICL) enables large language models to perform few-shot learning by conditioning on labeled examples in the prompt. Despite its flexibility, ICL suffers from instability -- especially as prompt length increases with more demonstrations. To address this, we treat ICL as a source of weak supervision and propose a parameter-efficient method that disentangles demonstration-induced latent shifts from those of the query. An ICL-based teacher generates pseudo-labels on unlabeled queries, while a student predicts them using only the query input, updating a lightweight adapter. This captures demonstration effects in a compact, reusable form, enabling efficient inference while remaining composable with new demonstrations. Although trained on noisy teacher outputs, the student often outperforms its teacher through pseudo-label correction and coverage expansion, consistent with the weak-to-strong generalization effect. Empirically, our method improves generalization, stability, and efficiency across both in-domain and out-of-domain tasks, surpassing standard ICL and prior disentanglement methods.
BibTeX
@inproceedings{jukic2025disentangling,
  author = {Josip Jukić and Jan Šnajder},
  title = {{Disentangling Latent Shifts of In-Context Learning with Weak Supervision}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2025},
  url = {https://openreview.net/forum?id=tAq9Gxdhr0},
}
Figure