Is the Meta-Learning Premium Paid for Memorization?

Meta-learning trains a model on many small tasks so that it can pick up a new task from a handful of examples. The most faithful version, called bi-level meta-learning, pays a high price for this. An inner loop adapts the model to each task, and an outer loop updates the model by differentiating through that adaptation. When MAML-en-LLM brought this procedure to GPT-2 Medium on 142 language tasks, training took about 12 hours, against 5 hours for plain multi-task fine-tuning (MetaICL). The payoff was roughly 2% on unseen domains and 4% in few-shot adaptation. Everything in that accounting assumes the unseen tasks really are unseen. Language models are pretrained on web text that often contains benchmark test sets, and Task Contamination showed that apparent few-shot skill frequently disappears on tasks the model could not have seen.

This note proposes a controlled test of whether the extra compute of bi-level meta-learning buys real adaptation ability or mostly buys faster recall of tasks already memorized during pretraining. The mechanism behind the suspicion is simple. The outer loop rewards the model most for tasks where one inner step produces a large drop in loss. A task whose format or content was memorized is exactly the kind of task that "snaps into place" after one step. So bi-level training may concentrate its gains on contaminated tasks, where contamination means the pretraining data already contained a benchmark's items or templates. We will plant contamination at controlled doses into pretraining and measure how much of the meta-learning premium survives on tasks that are clean by construction.

Core Claim

The central hypothesis is that the bi-level meta-learning premium over multi-task fine-tuning grows with pretraining contamination of the held-out tasks. The quantity we estimate is a difference-in-differences:

$$\Delta(c) = \big[A_{\text{bi}}(c) - A_{\text{mt}}(c)\big] - \big[A_{\text{bi}}(0) - A_{\text{mt}}(0)\big]$$

Here $A_{\text{bi}}(c)$ and $A_{\text{mt}}(c)$ are held-out task accuracies of bi-level meta-training and multi-task fine-tuning, and $c$ is the contamination dose of those tasks, with $c=0$ meaning clean. A reliably positive, dose-increasing $\Delta$ would mean part of the reported premium is memorization being reactivated. A $\Delta$ near zero would be an equally useful result. It would give the first contamination-controlled evidence that the premium is real, which would justify the compute toward the larger-scale bi-level training that the field wants but cannot yet afford.

This differs from the closest work in specific ways. Task Contamination audits prompting-based few-shot learning and instruction tuning with chronological splits and membership inference. It never examines gradient-based meta-training, and it never plants contamination causally. MAML-en-LLM measures the compute and accuracy trade-off, but it treats disjoint train and test task splits as proof of clean evaluation. Our study joins the two. It reads the compute overhead reported by the second paper through the contamination lens of the first, using an intervention rather than an observational audit.

Research Program

The base model is GPT-2 Medium. We use it to stay comparable with MAML-en-LLM and because it is the largest scale at which that paper's bi-level procedure has been shown to run.

We first build contamination arms by continued pretraining on a fixed token budget sampled from FineWeb. Into that sample we insert material from a randomly chosen subset of held-out CrossFit tasks, using the task collection and code from INK-USC/CrossTaskMoE. The inserted material comes in three forms. The first is verbatim test instances. The second is format-only contamination: the task's prompt template and label vocabulary filled with different content, which is the kind of leakage Li and Flanigan call task contamination. The third is paraphrased instances, which matters because Test of Time showed that rephrasing erases the usual detection signals while preserving what the model learns. Each form is inserted at three doses, set by the number of repetitions. A matched arm receives the same continued pretraining with no insertion, so that token count and optimizer state are identical across arms.

On each resulting checkpoint we then run MetaICL-style multi-task fine-tuning and MAML-en-LLM, including its shared optimizer moments. Held-out tasks fall into three groups: planted tasks, unplanted tasks from the same collection, and tasks that are clean by construction.

The clean-by-construction group matters because GPT-2's original pretraining may already contain older CrossFit datasets. That background contamination would otherwise blur the $c=0$ baseline. We build this group in two ways. The first converts DyVal problems into few-shot classification tasks. DyVal generates reasoning problems from randomly drawn directed acyclic graphs (DAGs, graphs with directed edges and no cycles) in boolean logic, reachability and arithmetic. The second uses probed rewrites in the style of Meta Probing Agents, which perturb benchmark items through choice permutation or inserted distractors while keeping their meaning. Among the unplanted CrossFit tasks, we also separate datasets released after GPT-2's training data was collected from older ones, and we check for overlap against an open reconstruction of GPT-2's training corpus.

Beyond measuring $\Delta$, we instrument training to test the proposed mechanism. For each task we log its share of the outer-loop gradient norm and its query-loss drop after one inner step. The mechanism predicts that planted tasks will dominate both.

We also borrow the effective-rank measure (ECI) from A Narrowing Geometry in Contaminated Reasoning. That paper found that contaminated models squeeze their activations into a few dominant "memorization directions". It also introduced Top-Subspace Suppression (TSS), which removes those directions at inference time. That work only studied standard supervised fine-tuning. We ask whether bi-level meta-training sharpens this narrowing more than multi-task fine-tuning does.

Finally, following On The Fragility of Benchmark Contamination Detection, we test a secondary question. That paper showed RL post-training hides membership signals. We ask whether meta-training on clean tasks similarly hides the planted contamination from standard loss-based and likelihood-ratio detectors, measured as detection quality before and after meta-training.

Baselines

The naive reference point is the continued-pretrained checkpoint prompted with in-context examples and no meta-training at all. It establishes how much of any held-out score contamination supplies for free. Any meta-learning gain must be read against it.

The strongest existing method, and the object of study, is MAML-en-LLM. It reports an average 2% gain on unseen domains, a 4% gain in adaptation, and a 2% gain with 10% of the training data over MetaICL on CrossFit and UnifiedQA, at 12 versus 5 hours of training. We found no reported result on Lacuna for MetaICL's or MAML-en-LLM's absolute CrossFit accuracies, so we will reproduce both on our unplanted arm first. We treat the reported 2 to 4 point relative gains as the reproduction target.

The most dangerous competitor is multi-task fine-tuning given the same compute as bi-level training: roughly 2.4 times the steps, or larger batches, with an equal hyperparameter search budget for each method. If this compute-matched version closes the gap on clean tasks, the premium was never about bi-level structure.

A first-order bi-level variant, which drops the second-order terms, isolates whether any contamination sensitivity comes from differentiating through the adaptation step. Applying TSS at evaluation isolates the memorization directions. Bi-level training "beats" these references only if its advantage on clean-by-construction and post-GPT-2 tasks exceeds seed variance, estimated over five seeds, and survives compute matching.

Why not just…?

Why not just evaluate on tasks published after the model's training cutoff? We do use such tasks, but on their own they are weak evidence. Test of Time shows that the drop in scores after a cutoff can vanish under simple rephrasing, and that tracing answers back to source documents falls from a 77.5% to a 17.5% hit rate once items are transformed. So the absence of a post-cutoff drop proves little. Few CrossFit-format tasks postdate GPT-2 in any case. Planted contamination gives a causal dose-response that a chronological split cannot.

Why not just filter held-out tasks by n-gram overlap with the pretraining data? Format-only and paraphrased contamination contain little verbatim overlap. That is exactly why Task Contamination needed extraction and membership tests beyond overlap. Our paraphrase and format arms directly measure how much of $\Delta$ an n-gram filter would miss.

Why not just meta-test on dynamic benchmarks like DyVal and skip real tasks? This is a genuine threat to the need for the planting design, because DyVal is clean by construction and cheap. But DyVal covers only formal domains, while the premium MAML-en-LLM reports is on natural-language tasks. A clean-synthetic result cannot tell us whether the premium on real tasks came from contamination. If the premium on DyVal-derived tasks matches the premium on planted-free CrossFit tasks, we will report that the simpler protocol suffices.

Why not just add task interpolation to regularize against memorization? Meta-Learning with Fewer Tasks through Task Interpolation (MLTI) mixes pairs of training tasks to create synthetic ones, which curbs memorization of the training tasks. Our problem is different: memorization of held-out tasks that happened during pretraining. Interpolation cannot remove memorization that happened before meta-training began, so we include it only as an optional arm.

Decisive Checks

The main test is the sign, and the change with dose, of $\Delta$ on planted versus unplanted tasks, averaged over five seeds and all three contamination forms. The mechanism holds if planted tasks carry a disproportionate share of the outer-loop gradient and show lower effective rank after bi-level training than after multi-task fine-tuning. A further check applies TSS: if suppressing the memorization directions removes the bi-level advantage on planted tasks while leaving it intact on clean ones, memorization is being reactivated.

Efficiency is reported as clean-task accuracy per GPU-hour. This produces a contamination-corrected compute frontier for bi-level meta-learning, unlike the width-scaling frontier studied for learned optimizers in μLO.

The hypothesis fails in one of two ways. If $\Delta$ stays near zero across doses while the compute-matched baseline still trails, the bi-level premium is genuine and the compute is justified. If compute-matched fine-tuning equals bi-level training on clean tasks, the premium is a compute artifact regardless of contamination.