In-Context Learning (ICL) has fundamentally changed how we interact with Large Language Models (LLMs). By simply prepending a few examples—or "shots"—to a prompt, a frozen model can instantly adapt to a new task without a single weight update.
However, this flexibility comes with a steep operational penalty known as the prompt tax. As models scale and tasks grow more complex, the computational and financial overhead of processing long context windows becomes a critical bottleneck for real-world deployment.
The Mechanics of the Prompt Tax
To understand why in-context learning is computationally expensive, we must look at how the Transformer's self-attention mechanism processes a prompt.
When you provide a model with multiple demonstrations, the total sequence length increases linearly with the number of examples. Because standard self-attention requires every token to attend to every other token, the computational complexity scales quadratically ($O(L^2)$) with the sequence length. As shown by Jiahao Li et al. in Feature-Adaptive and Data-Scalable In-Context Learning, this quadratic scaling causes memory consumption and inference latency to explode when moving toward "many-shot" regimes. This issue is magnified in Large Multimodal Models (LMMs), where a single image can consume hundreds of tokens, making visual examples incredibly expensive (Brandon Huang et al. in Multimodal Task Vectors Enable Many-Shot Multimodal In-Context Learning).
Furthermore, ICL is inherently ephemeral. In a standard setup, the model must re-process the entire set of demonstrations for every new query. While Key-Value (KV) caching can preserve the representations of static prompts across queries, Rishabh Agarwal notes in Many-Shot In-Context Learning that maintaining massive KV caches for thousands of tokens introduces severe memory management challenges. If the prompt is dynamically tailored to the query—such as retrieving semantically similar examples to maximize accuracy—the KV cache cannot be reused at all, forcing a costly re-computation of the hidden states for every single inference call (Katerina Margatina et al. in Active Learning Principles for In-Context Learning with Large Language Models).
To bypass these limitations, researchers are developing methods to compress, internalize, or restructure context so that models can retain the adaptability of ICL at a fraction of the cost.
Compressing Context into Latent Task Vectors
One insight into why ICL works is that the model's attention layers naturally compress the provided demonstrations into a steady-state direction in the latent space. This representation is often called a task vector or function vector. As Yuxin Dong et al. demonstrate in Understanding Task Vectors in In-Context Learning: Emergence, Functionality, and Limitations, these vectors act as a distilled essence of the task.
If the model automatically distills demonstrations into a vector, we can skip passing the raw tokens entirely. Instead, we can isolate this vector and inject it directly into the model's internal activations.
Standard ICL:
[Example 1] + [Example 2] + [Example 3] + [Query] ---> [Transformer Layers] ---> Output
(High token overhead, quadratic attention cost)
Task Vector Injection:
[Query] + 注入(Task Vector) -------------------------> [Transformer Layers] ---> Output
(Zero token overhead, constant attention cost)
This mechanism forms the basis of several efficiency frameworks:
- Cross-Modal Patching: Grace Luo et al. show in Vision-Language Models Create Cross-Modal Task Representations that VLMs build unified task representations across modalities. By calculating a task vector from image-text pairs and "patching" it directly into the latent space, they trigger specific behaviors without the overhead of long contextual sequences. Alberto Hojel et al. apply a similar principle in Finding Visual Task Vectors to eliminate redundant image processing.
- Soft Injection: Rather than appending text tokens, Jungwon Park et al. argue in Soft Injection of Task Embeddings Outperforms Prompt-Based In-Context Learning that shifting task conditioning directly into the model's internal activations preserves KV cache memory and significantly reduces latency while maintaining or exceeding few-shot accuracy.
- Discrete Compression: If text prompts must be used, they can be compressed algorithmically. Leyang Cui et al. develop a self-supervised method called Selection-$p$ in Selection-p: Self-Supervised Task-Agnostic Prompt Compression for Faithfulness and Transferability, which compresses prompts up to 10x in a single forward pass by filtering out non-essential tokens while maintaining transferability across models.
Amortizing Context into Weights and Adapters
Another approach shifts the context out of the activation space entirely and bakes it directly into the model's parameters. This turns a per-token inference burden into a one-time weight adjustment.
Hypernetworks and Online Generation
Instead of running an expensive fine-tuning loop over demonstrations, a separate, lightweight network can read the context and instantly generate parameter updates—such as Low-Rank Adaptation (LoRA) weights—in a single forward pass.
[Task Description / Context] ---> [Hypernetwork] ---> Task-Specific LoRA Weights
|
v
[User Query] -------> [Frozen LLM] -------> Output
This strategy allows for instant model specialization. Rujikorn Charakorn et al. introduce this in Text-to-LoRA: Instant Transformer Adaption, allowing an LLM to adapt to a task instantly based on a natural language description. Similarly, Yicheng Xiao et al. propose LoRA-Gen in LoRA-Gen: Specializing Large Language Model via Online LoRA Generation, using a cloud model to generate weights for edge devices, achieving over 10x prompt compression. For long-context document memorization, Tong Chen et al. design the Generative Adapter: Contextualizing Language Models in Parameters with A Single Forward Pass, which maps long documents directly into parameter space to eliminate KV cache growth during generation.
Exact Mathematical Conversion
For specific architectures, we can bypass hypernetworks entirely. Brian K Chen et al. demonstrate in Exact Conversion of In-Context Learning to Model Weights in Linearized-Attention Transformers that in Linearized-Attention Transformers, an ICL prompt can be mathematically converted into exact model weights by adding specific bias terms. This makes temporary context permanent and computationally free at inference time.
Knowledge Distillation
When using ICL for tasks like knowledge editing (updating facts in a model via prompts), the prompt length grows with every new fact. To fix this, Shanbao Qiao develops DistillMIKE in DistillMIKE: Editing Distillation of Massive In-Context Knowledge Editing in Large Language Models, which uses a distillation process to bake massive in-context updates directly into LoRA weights, eliminating the severe inference-time slowdowns documented in prompt-based knowledge editing (Ce Zheng et al. in Can We Edit Factual Knowledge by In-Context Learning?).
Architectural and Cache-Centric Optimizations
If the raw context must be preserved, we can alter the underlying architecture or cache mechanics to process it more efficiently.
Decoupled Encoding (Fusion-in-Decoder)
In a standard Transformer, all demonstrations are concatenated into a single string, meaning tokens in Example 3 compute attention over tokens in Example 1 and Example 2. This cross-example attention is largely redundant.
Qinyuan Ye et al. exploit this insight in FiD-ICL: A Fusion-in-Decoder Approach for Efficient In-Context Learning. By encoding each demonstration independently and combining them only in the decoder, they break the joint quadratic bottleneck. If you have 4 examples of length 100, standard attention scales as $(4 \times 100)^2 = 160,000$ operations, whereas decoupled encoding scales as $4 \times (100)^2 = 40,000$ operations—resulting in a 10x inference speedup.
Standard Attention:
[ Ex 1 <---> Ex 2 <---> Ex 3 <---> Query ] --> Quadratic explosion across all tokens
Fusion-in-Decoder Attention:
[ Ex 1 ] ---\
[ Ex 2 ] ----+--> [ Decoder merges representations ] <---> [ Query ]
[ Ex 3 ] ---/
(No attention computed between different examples)
KV-Cache Retrieval
For personalized applications where a model must reference a user's history or metadata, Binxiao Xu introduces Jarvis in Jarvis: Towards Personalized AI Assistant via Personal KV-Cache Retrieval. Instead of forcing the model to re-parse background information for every interaction, Jarvis precomputes and stores user-specific text and image patches directly as KV-cache states, retrieving and injecting them dynamically to achieve up to a 10x improvement in throughput.
Parameter-Efficient Fine-Tuning as an Alternative
Ultimately, some researchers argue that trying to optimize ICL is fundamentally less efficient than abandoning it for specialized tasks.
Mohit Bansal et al. demonstrate this in Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning. They introduce the "T-Few" recipe using $(IA)^3$, a parameter-efficient fine-tuning (PEFT) method that scales down the model size requirement. By updating a tiny fraction of the weights on the few-shot data, they outperform massive models like GPT-3 175B while being 1,000x cheaper to run at inference time because the input prompts remain short.
Similarly, Boyan Gao et al. propose Optimization-Inspired Few-Shot Adaptation (OFA) in Optimization Inspired Few-Shot Adaptation for Large Language Models. By treating the forward pass as a sequence of gradient descent steps and tuning only LayerNorm parameters, they achieve strong task generalization without consuming the context window or incurring the per-token inference tax of ICL.
Key Foundational Work
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022) — contributed 1 concept elements to this direction.
Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning (2022) — contributed 1 concept elements to this direction.
HyperTuning: Toward Adapting Large Language Models without Back-propagation (2022) — contributed 1 concept elements to this direction.