Robot policies trained by imitation, meaning they copy recorded human demonstrations, generalize better when trained on many different tasks than when trained on many copies of a few. Language and reinforcement-learning models trained on synthetic task families show something sharper. Below a critical number of distinct tasks they memorize, and above it they abruptly switch to a general solution. Robotics has not asked whether the same switch happens there, and the question has a trap in it. Long-horizon success is a product of per-step successes, so a smooth improvement in each step can look like a sudden jump in whole-episode success. Any "threshold" found by plotting episode success against task count may be nothing more than compounding arithmetic.
This note proposes to take that confound apart and to test a specific mechanism. The hypothesis is that task diversity mostly changes whether a policy pulls itself back after drifting off the demonstrated path, rather than how accurate it is on that path. We will sweep pretraining task count at fixed total data on RoboCasa365. At each point we measure three things: per-step accuracy, closed-loop recovery from injected perturbations, and whether the policy on unseen tasks just replays the nearest training task. Recovery data is a cheap way to buy self-correction, so we also ask how many tasks' worth of diversity a small amount of noise-injected recovery data is worth.
Core Claim
The claim has three parts.
First, as the number of distinct pretraining tasks grows, a visuomotor policy's closed-loop contraction changes much more sharply than its per-step imitation error. Contraction means the rate at which a perturbed state returns toward states consistent with the task.
Second, long-horizon success on held-out composite tasks departs from what per-step accuracy alone predicts, and the departure appears where contraction changes.
Third, the change resembles the memorization-to-generalization transitions seen in synthetic settings. Below it, policies on unseen tasks reproduce the trajectories of the nearest training task; above it, they do not.
If the first two parts fail, the robotics "threshold" is a product-rule artifact. That is also worth establishing, because it would redirect effort from task breadth toward per-step accuracy.
The synthetic literature motivates this picture. Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression shows linear-regression Transformers switch from a prior-matching estimator to ridge regression at roughly $2^{14}$ to $2^{15}$ tasks for dimension 8. It also shows weight decay lowers that threshold. Competition Dynamics Shape Algorithmic Phases of In-Context Learning shows that low diversity favors retrieval of memorized training chains, and that larger models retrieve more readily. Differential learning kinetics govern the transition from memorization to generalization during in-context learning explains the switch as a speed race between a memorizing circuit and a generalizing circuit. It also finds the generalizing solution can be transient under long training. Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds finds a similar switch near 10,000 tabular decision problems. All four use discrete or synthetic task families with a clean task count and no continuous closed-loop dynamics. We borrow their diagnostics, not their settings.
The robotics side documents the need but not the mechanism. RoboCasa365 reports that moving from 50 to 300 pretraining tasks helps more than adding synthetic data to fewer tasks, and that composite tasks with 15 or more sub-steps remain very hard. It tests only coarse increments and does not separate per-step error from error growth. Data Scaling Laws in Imitation Learning for Robotic Manipulation finds smooth power laws in environment and object diversity, with saturation near 50 demonstrations per pair. It is explicitly single-task. BridgeData V2 and DROID show that breadth beats depth at fixed volume, but neither measures how errors grow along a rollout.
The compounding-error theory supplies our instruments. Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces makes two points relevant here. Executing actions in open-loop chunks tames error growth when the underlying plant is stable. Injecting noise during data collection, while recording the expert's clean corrective actions, tames it otherwise. That paper is single-task. We treat its noise injection as the direct competitor to task diversity for producing contraction.
Research Program
Task-count sweep. RoboCasa365 gives us 365 tasks with atomic/composite structure and 2,500 scenes. We hold out a set of test atomic tasks and a set of composite tasks whose component skills are only partly covered in pretraining. From the remaining pool we build nested pretraining subsets of increasing task count, spaced geometrically from about 8 to 300. The total number of demonstration frames is held fixed across subsets, so that breadth and depth are traded directly, as in BridgeData V2 and DROID. Subsets are drawn several ways per size, so that task identity does not masquerade as task count.
Pretrain–Test Task Alignment Governs Generalization in In-Context Learning shows that misaligned diversity can raise test error. To account for this, we also build a second family of subsets that grows only within manipulation families unrelated to the held-out tasks, such as appliance operation versus pick-and-place. This constrains the claim: added diversity should help only when it is aligned with the test tasks.
Policy. The primary policy is Diffusion Policy with action chunks, trained at each point with several seeds. We keep checkpoints throughout training to catch transience. GR00T N1.5 fine-tuning is run at a few sweep points as a high-capacity reference.
Three diagnostics, computed on held-out tasks after a small, fixed amount of target data.
Per-step accuracy. We measure the success probability $p_i$ of each sub-step when the policy starts from demonstration states. This defines the null prediction for a composite task of $n$ sub-steps:
$$\hat S_{\text{null}}(n) = \prod_{i=1}^{n} p_i .$$
Closed-loop contraction. Mid-rollout, we displace the end-effector or object by a controlled amount $d_0$. We then measure the distance $d_k$ from the policy's state after $k$ steps to the nearest state on successful demonstrations of that task, and define
$$\rho = \left(\frac{d_k}{d_0}\right)^{1/k}.$$
Values of $\rho$ below 1 mean the policy pulls itself back; values above 1 mean it drifts further. We report $\rho$ and the recovery rate as functions of task count and perturbation size.
Retrieval score. This is the robotic analogue of the retrieval-versus-inference decomposition in Competition Dynamics. For each held-out rollout, we compare its trajectory's distance to the nearest training-task demonstration against its distance to the correct held-out demonstration. A high retrieval score means the policy is replaying a memorized task.
As a secondary signal, we track the effective rank of the policy's hidden states. Mitigating Compounding Error via Video Representation Regularization found that drops in this rank precede drift in video rollouts, and that more data without regularization made them worse. Checking whether the same holds in closed-loop control tests their "scaling paradox" against the robotics scaling results.
Diversity versus recovery data. At each task count, we also train a variant whose pretraining data includes a matched share of noise-injected recovery segments with clean corrective labels. We generate these through RoboCasa's demonstration-generation tooling, following the recipe in the Action Chunking paper. If contraction is what diversity buys, recovery data should move the transition to fewer tasks, much as weight decay lowers the regression threshold. The size of that shift is the exchange rate between recovery data and task breadth.
Perturbation stress test. Final checkpoints are evaluated under visual and physical perturbations modeled on THE COLOSSEUM: lighting, distractors, object color, mass and friction. COLOSSEUM reports 30 to 50% drops under isolated perturbations and over 75% failure under combined ones. We will test whether the combined-perturbation drop shrinks faster with task count than the isolated drop. That pattern is what one would expect if diversity mainly buys recovery rather than invariance.
Baselines
Depth scaling. The naive reference spends the entire fixed frame budget on the smallest task subset, with more demonstrations per task. BridgeData V2, DROID and the Data Scaling Laws saturation result predict it loses on held-out tasks. The comparison still matters because it tells us whether depth scaling improves $\rho$ on the training tasks without transferring it. It uses the identical architecture, frame budget, training steps and target fine-tuning data.
Strongest existing system on the benchmark. RoboCasa365 reports GR00T N1.5 at a 51.1% average success rate in its pretrain-then-fine-tune setting. We fine-tune GR00T N1.5 on our largest aligned subset and on a mid-sized one, using the same held-out task split and target data, and budget-matched hyperparameter search. Our goal is not to beat 51.1% with Diffusion Policy. The comparison tests whether a high-capacity model's transition appears at fewer or more tasks than Diffusion Policy's. Competition Dynamics predicts that larger models retrieve more readily, so the transition should appear later. Beating this reference means one of two things: our recovery-augmented Diffusion Policy matches GR00T's held-out composite success at a smaller task count, or GR00T's retrieval score is measurably higher at matched task count.
Strongest compounding-error method. Chain-of-Action reports 55.2% on RLBench-60 against 38.9% for ACT and 32.6% for Diffusion Policy. Under spatial extrapolation it reports 48% against 8% and 4% (Chain-of-Action; code at ByteDance-Seed/Chain-of-Action). We run it on RLBench-60 at three task-count levels, taken as nested subsets of the 60 tasks, with the same data per task as Diffusion Policy. This tests whether goal anchoring flattens the diversity curve, which would mean architecture substitutes for breadth.
Ablations. Three ablations isolate the key ingredients. The first compares chunk lengths from 1 to 50 at fixed task count, to separate open-loop stabilization from diversity-driven contraction. The second is the recovery-data variant without added tasks. The third evaluates the null model $\hat S_{\text{null}}$ on its own; it is the analysis ablation that decides whether any observed threshold is real.
Why not just…?
Why not just tune the action chunk length? Chunking provably stabilizes only open-loop-stable plants (Action Chunking and Data Augmentation). Chunked ACT and Diffusion Policy fall to 8% and 4% under spatial extrapolation in Chain-of-Action. Our chunk-length ablation measures $\rho$ directly. If longer chunks give low-diversity policies the same contraction and held-out composite success as high-diversity ones, the diversity story is wrong for this benchmark, and we will say so.
Why not just use a bigger pretrained visual encoder? Data Scaling Laws found encoder scaling helps but does not remove the need for environment diversity. COLOSSEUM's pretrained-representation models still lost 30 to 50% under isolated perturbations. Encoders plausibly buy visual invariance but not recovery from physical displacement. We check this by running the sweep with two encoder sizes at the endpoints and comparing their effect on $\rho$ under physical perturbations. If the larger encoder alone closes the gap there, the claim narrows to visual shift.
Why not just add recovery data by noise injection and skip diversity? This is the real threat, and the proposal is built to measure it rather than dismiss it. If noise-injected recovery data on few tasks matches many-task pretraining on held-out tasks, the practical conclusion is to collect recovery data. We expect it not to transfer fully to unseen tasks, because it corrects around trained trajectories and does not stop the policy from retrieving the wrong task. The retrieval score distinguishes these two cases.
Why not just add a language-model planner? A planner shortens the horizon of each low-level segment but does not fix drift within contact-rich segments. ChainedDiffuser reports motion-planner-linked macro-actions (Act3D) at 21.0% on continuous-interaction RLBench tasks, against 80.9% for continuous trajectory diffusion (ChainedDiffuser). Our atomic held-out tasks isolate this segment-level question.
Proposed Experiments and Decisive Checks
The first decisive check is the null test. For each task count, we compare held-out composite success against $\hat S_{\text{null}}(n)$ computed from measured sub-step successes. If observed success tracks the null curve across the sweep, any apparent threshold is compounding arithmetic, and the mechanistic claim is rejected. The claim survives only if observed success exceeds the null model by a margin outside the seed-to-seed spread, starting at a task count where $\rho$ crosses below 1.
The second check concerns sharpness. Differential learning kinetics predicts bimodal behavior across seeds near a true transition. We therefore fit smooth power-law and sigmoidal forms to $\rho$ and to the retrieval score versus log task count, and examine the across-seed distribution at each point.
The third check is the recovery-data shift. We report the task count at which $\rho$ first drops below 1, with and without recovery segments. A leftward shift supports the contraction mechanism; no shift weakens it.
The fourth check is alignment. Diversity added from unrelated manipulation families should move $\rho$ and held-out success less than aligned diversity, and possibly harm them, as the alignment paper predicts.
The fifth check is transience. We examine late checkpoints for the retrieval score creeping back up.
The sixth check is transfer across benchmarks. The RLBench-60 subset sweep with Diffusion Policy, ACT and Chain-of-Action tests whether the same pattern appears on a second benchmark and how goal anchoring changes it. The file gives only the 50-versus-300 comparison on RoboCasa365 and no reported per-step error growth rates. For the growth rate of error on RoboCasa365 composite tasks, no reported result was found on Lacuna, so our null-model decomposition will be the first such measurement.
References
- Pretraining task diversity and the emergence of non-Bayesian in-context learning for regression
- Competition Dynamics Shape Algorithmic Phases of In-Context Learning
- Differential learning kinetics govern the transition from memorization to generalization during in-context learning
- Towards Large-Scale In-Context Reinforcement Learning by Meta-Training in Randomized Worlds
- RoboCasa365: A Large-Scale Simulation Framework for Training and Benchmarking Generalist Robots
- Data Scaling Laws in Imitation Learning for Robotic Manipulation
- BridgeData V2: A Dataset for Robot Learning at Scale
- DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset
- Action Chunking and Data Augmentation Yield Exponential Improvements in Behavior Cloning for Continuous Spaces
- Pretrain–Test Task Alignment Governs Generalization in In-Context Learning
- Mitigating Compounding Error via Video Representation Regularization
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
- Chain-of-Action: Trajectory Autoregressive Modeling for Robotic Manipulation
- ChainedDiffuser: Unifying Trajectory Diffusion and Keypose Prediction for Robotic Manipulation