TL;DR
In-Context Learning (ICL) allows large language models to learn from a few examples in a prompt, but it suffers from high inference costs and extreme sensitivity to example ordering. This paper introduces WILDA, a method that treats ICL as a source of weak supervision to train a lightweight LoRA adapter on unlabeled queries, effectively "baking" the demonstrations directly into the model's weights. Surprisingly, the resulting student model not only eliminates the need for long prompts but actually outperforms its ICL teacher through weak-to-strong generalization.
What Problem Does This Paper Address?
Large Language Models (LLMs) are famous for their ability to perform few-shot learning via In-Context Learning (ICL)—simply prepend a few labeled examples (demonstrations) to a query, and the model figures out the task without any weight updates.
However, ICL has two major flaws:
- Instability: LLMs are notoriously fickle. Changing the choice or even just the ordering of the demonstrations can cause accuracy to swing wildly.
- Inefficiency: Prepending examples forces the model to process long contexts for every single query. Because transformer attention scales quadratically with sequence length, this inflates compute costs, introduces positional biases, and eats up the model's context window.
Prior attempts to solve this tried to "disentangle" the effect of demonstrations from the query by hacking into the model's internal attention heads or hidden states. However, these methods rely on simplified linear approximations of attention that fail to capture the complex, non-linear dynamics of real-world transformers.
Methods
The authors propose WILDA (Weakly-supervised In-context Learning Disentanglement via Adapters). Instead of manipulating internal mechanics, WILDA takes a functional approach using a teacher-student framework:
- The Teacher: A standard LLM is prompted with the few-shot demonstrations alongside an unlabeled query to generate a "pseudo-label" (the full probability distribution over the vocabulary).
- The Student: The exact same LLM, but equipped with a lightweight LoRA adapter, is given only the query (no demonstrations). It is trained to match the teacher's output distribution by minimizing cross-entropy loss.
By doing this, the adapter internalizes the latent shift induced by the demonstrations. The authors experiment with three training variants, finding that shuffling the demonstration order between training epochs (WILDA-S) yields the highest stability. Furthermore, because the knowledge is stored in modular adapters, multiple adapters trained on different demonstration subsets can be combined using simple adapter arithmetic (adding their weights together).
Datasets
The method was evaluated using Llama 3 (8B), Phi 3 (mini 4k), and Llama 2 (7B) across several standard benchmarks:
- GLUE Benchmark: 7 tasks covering single-sequence and sequence-pair classification (RTE, SST, QNLI, MNLI, COLA, MRPC, QQP).
- MMLU Benchmark: Selected subsets including "elementary math" (MATH) and "miscellaneous" (MISC).
- ARC-Challenge: Used to evaluate more demanding multi-hop reasoning.
For training, WILDA only required a small pool of 100 to 500 unlabeled instances per task to act as queries for the teacher.
Key Insights and Ideas
- Weak-to-Strong Generalization: The most fascinating finding is that the student regularly surpasses the teacher. Even though the teacher's pseudo-labels are noisy and inconsistent, the student filters out this noise through pseudo-label correction (enforcing local consistency in the representation space) and coverage expansion (generalizing beyond the narrow scope of the few demonstrations).
- Smoother Loss Landscapes: Empirical analysis showed that WILDA significantly lowers the model's approximated Lipschitz constant compared to standard ICL. In plain English: the model becomes much less sensitive to small perturbations in the input, explaining its massive boost in stability.
- True Disentanglement: By testing the student's ability to reconstruct the original demonstrations word-for-word from the adapter weights alone, the authors proved that the adapters faithfully and cleanly encode the specific task context.
Key Results
- Superior Accuracy: On Llama 3 (8B), WILDA-S outperformed standard ICL across all datasets, yielding absolute accuracy improvements between 2.6% and 11.9%.
- Unmatched Stability: When shuffling the order of 16 demonstrations 50 times, standard ICL showed high variance (e.g., a standard deviation of 4.81 on the RTE dataset). WILDA-S crushed this instability, bringing the standard deviation down to 0.22.
- Amortized Efficiency: While training an adapter incurs an upfront cost, removing demonstrations speeds up inference drastically. The authors calculated that for a 16-shot setup on Llama 3, the computational cost of fine-tuning breaks even after roughly 2,100 inferences, after which every query is pure profit in terms of speed and compute savings.
Limitations and Future Work
- Data & Compute Overhead: WILDA is not a zero-cost solution. It requires a small set of unlabeled data and an initial fine-tuning phase (10 epochs), making it less ideal for scenarios where you only need to process a handful of queries.
- Massive Demonstration Scales: While the authors successfully fused up to 8 adapters (encoding 128 total demonstrations), the behavior and scalability of WILDA when dealing with thousands of demonstrations remains unexplored.
Summary updated