Abstract
This paper introduces context-enhanced learning (CEL), a training paradigm where Large Language Models (LLMs) perform gradient-based learning on a task while privileged reference material is provided in the prompt. Crucially, no gradients are computed on the reference material itself. We demonstrate that for complex, multi-step reasoning tasks, CEL is exponentially more sample-efficient than standard supervised fine-tuning (SFT). Through empirical evaluation on synthetic benchmarks and theoretical analysis using surrogate models, we show that CEL allows models to internalize knowledge into their weights, effectively bridging the gap between in-context learning and weight-based capability.
Introduction
Large Language Models (LLMs) exhibit remarkable In-Context Learning (ICL) abilities, solving novel tasks via few-shot prompting. However, Supervised Fine-Tuning (SFT) remains the standard for permanent capability integration. SFT is notoriously sample-inefficient for deep reasoning tasks because the model must discover the correct "gradient path" through a high-dimensional space of potential solutions.
This research investigates whether ICL can serve as a catalyst for weight-based learning. By providing a "textbook" or privileged context during training—without requiring the model to memorize the text itself—can we accelerate the internalization of the underlying logic? We formalize this as Context-Enhanced Learning and explore its efficiency, mechanistic properties, and privacy implications.
Methodology
The core of our approach is Context-Enhanced Learning. During the training phase, the model receives an input pair consisting of [Curriculum-Text, Task-Input]. The model is tasked with predicting the Task-Output, but the auto-regressive loss is calculated exclusively on the output tokens. The "Curriculum-Text" (e.g., rules, definitions, or phrasebooks) provides the necessary information to solve the task via ICL, thereby providing a high-quality gradient signal for the task output.
To ensure the model internalizes the logic rather than becoming dependent on the prompt, we employ two primary techniques:
- Annealing Dropout: The probability of omitting the curriculum text is gradually increased from 0% to 100% during training. This forces the model to transition from relying on external context to utilizing its internal weights.
- Silent Chain-of-Thought (CoT): The model generates a sequence of
<THINK>tokens before the final answer. This provides internal computational steps that the model can use to handle multi-step reasoning at test time without explicit prompting.
Experimental Setup
We evaluate CEL using the Multi-layer Translation (MLT) benchmark. This synthetic task requires a bijective mapping between $d+1$ different languages. To translate from the initial to the final language, a model must sequentially apply $d$ distinct "phrasebooks" (bigram transformation rules) interspersed with circular shifts.
The complexity of the task is defined by the number of characters $n$ and the depth $d$. Each output character depends on $2d$ input characters. We utilized Llama 3.2-3B as our base model, pre-conditioning it to ensure basic ICL competence on the MLT task before initiating CEL experiments.
Key Results
Our experiments demonstrate a massive leap in sample efficiency. As shown in Figure 5, for a task with $d=5$ layers, the "Annealing Dropout" strategy achieves perfect test accuracy with approximately $10^5$ samples. In contrast, the standard SFT baseline ("No Context") fails to exceed random guess accuracy even with $10^6$ samples in the $n=10$ configuration.
Mechanistic Insights
- Gradient Quality: The presence of context provides a "cleaner" gradient. Instead of guessing intermediate steps, the model follows the path provided in-context, allowing gradients to update weights directly for each sub-step.
- Layer Specialization: Mechanistic analysis reveals that specific transformer layers learn to handle specific translation steps (e.g., Layer 10 for Language 1→2). CEL facilitates this local specialization.
- Atomic Internalization: The model learns rules "atomically," enabling it to generalize to new permutations of the task at test time.
Discussion and Conclusion
Context-Enhanced Learning offers a path toward more efficient LLM training for complex reasoning. Theoretically, we prove that while SFT requires $n^{\Omega(d)}$ samples (exponential in depth), CEL requires only $O(poly(n)d \log d)$ samples (polynomial).
Furthermore, CEL appears to be "stealthy." The model internalizes the logic of the privileged context without memorizing the literal strings, making it difficult to recover the training "textbook" through likelihood queries. While the synthetic nature of MLT is a limitation, these results suggest that CEL could significantly reduce the data requirements for teaching LLMs sophisticated, multi-step reasoning capabilities.
Summary updated