Abstract
In-context learning (ICL) has become essential in natural language processing, particularly with autoregressive large language models capable of learning from demonstrations provided within the prompt. However, ICL faces challenges with stability and long contexts, especially as the number of demonstrations grows, leading to poor generalization and inefficient inference. To address these issues, we introduce STICL (Self-Training ICL), an approach that disentangles the latent shifts of demonstrations from the latent shift of the query through self-training. STICL employs a teacher model to generate pseudo-labels and trains a student model using these labels, encoded in an adapter module. The student model exhibits weak-to-strong generalization, progressively refining its predictions over time. Our empirical results show that STICL improves generalization and stability, consistently outperforming traditional ICL methods and other disentangling strategies across both in-domain and out-of-domain data.
Introduction
In-context learning (ICL) allows Large Language Models (LLMs) to perform new tasks by simply observing a few examples in a prompt. Despite its power, ICL is plagued by three primary "headaches":
- Instability: Performance can fluctuate wildly based on the specific selection or ordering of examples.
- Inefficiency: Every query requires reprocessing the entire list of demonstrations, consuming memory (KV cache) and slowing inference.
- Scaling Limits: Context windows restrict the number of examples a model can process before becoming confused or running out of space.
This paper introduces WILDA (Weakly-supervised In-context Learning Disentanglement via Adapters), a method that addresses these issues by treating ICL as a "weak teacher" to train a lightweight "student" adapter.
Methodology
WILDA disentangles the task-specific knowledge contained in demonstrations from the query itself through a functional self-training workflow.
As shown in Figure 1, the process follows these steps:
- The Teacher: An LLM is prompted with standard ICL (demonstrations + query) to generate "pseudo-labels" on a small set of unlabeled data.
- The Student: The same LLM, now without demonstrations in the prompt, is equipped with a LoRA adapter. It is trained to match the teacher's predictions using only the query as input.
- Disentanglement: Because the student only sees the query, the adapter is forced to capture the "latent shift"—the specific task logic—that the demonstrations originally provided.
A standout feature of WILDA is its composability via "Adapter Arithmetic." Multiple adapters can be trained on different subsets of data and their weights can be added together to fuse knowledge, effectively bypassing context window limits.
Experimental Setup
The researchers evaluated WILDA across several diverse benchmarks using models like Llama 3 (8B) and Phi 3 (mini):
- GLUE: Seven tasks covering sentiment analysis, linguistic acceptability, and natural language inference (e.g., SST, MNLI, QQP).
- MMLU: Specifically the "elementary math" and "miscellaneous" subsets.
- Unlabeled Data: Small pools of 100 to 500 unlabeled instances were used to generate the teacher's pseudo-labels.
Key Results and Discussion
Empirical results show that WILDA consistently outperforms traditional ICL methods and other structural disentanglement strategies.
Weak-to-Strong Generalization
A significant finding is that the student often surpasses the teacher. As seen in the results for Llama 3 (Figure 2), WILDA-S (the shuffled variant) improved absolute accuracy by 2.6% to 11.9% over standard ICL. This "weak-to-strong" phenomenon occurs because the student smooths out inconsistent teacher errors (pseudo-label correction) and generalizes the task logic to a broader range of inputs (coverage expansion).
Stability and Efficiency
WILDA provides rock-solid stability. While standard ICL is sensitive to example ordering, moving task knowledge into parametric adapter weights significantly reduces performance variance. Furthermore, WILDA offers massive inference speedups. For a 16-shot setup, standard ICL can be ~120x slower than a 0-shot query; WILDA provides many-shot performance at 0-shot speeds.
Conclusion
WILDA effectively transforms transient in-context information into reusable parametric knowledge. By disentangling latent shifts through self-training, it overcomes the inherent instability and inefficiency of traditional ICL. While there is an upfront cost for training, the method breaks even after approximately 2,100 inferences, making it a robust solution for deploying ICL-based tasks at scale.
Summary updated