Spending the Simulator Where the Camera Is Blind

Robots that learn from simulated demonstrations pay twice for every example. They pay once to plan and render a trajectory, and again to simulate contact physics if the policy is meant to use touch. When one policy is trained on many tasks, the usual worry is that tasks fight over shared network weights. This proposal argues that, in multi-task visuo-tactile imitation from simulation, the scarcer resource is generation compute, not gradient memory. Tasks should therefore be balanced where the data is made, not only where losses are summed. We propose a lightweight controller that decides, after each round of training, how many new synthetic demonstrations to generate for each task and which part of the trajectory they should cover. It keeps only a few scalar loss statistics per task, in the spirit of loss-history balancing methods. Its main lever is the moment of visual occlusion. When the gripper closes on an object, the camera stops seeing the contact, and touch carries the information. Those terminal windows are short, expensive to simulate with tactile physics, and where multi-task policies fail. The controller can restart the simulator from saved pre-contact states and pour samples into exactly those windows.

The opportunity is concrete. Real visuo-tactile datasets are costly: DECO needed 50 hours of humanoid teleoperation, and Touch in the Wild needed over 2,700 handheld demonstrations. Fast tactile simulation now exists. Efficient Tactile Simulation with Differentiability simulates normal and shear contact fields at 1000 FPS on one CPU core and transfers zero-shot to a real GelSlim peg insertion. The open question is how to spend a fixed simulation budget across many tasks and phases so that a single policy with touch improves fastest.

Core Claim

Given a fixed total generation budget, measured in CPU/GPU-seconds and counting rejected and colliding trajectories, a multi-task visuo-tactile policy trained on data allocated by a per-task, per-phase progress controller will reach higher average success than two alternatives at the same budget. The first is uniform per-task data generation. The second is uniform generation combined with adaptive loss reweighting. The gain should come mostly from occluded contact phases. Two simpler explanations could account for any gain, and the design must rule them out. One is "it is just loss reweighting in disguise." The other is "it is just more data on hard tasks, regardless of phase."

How This Relates to Prior Work

Three groups of papers bear on the claim.

The multi-task optimization literature supplies the scalar-statistics method, and some of it constrains the design. FAMO reweights task losses so that all task log-losses fall at a similar rate. It needs only one backward pass and constant extra memory, and it beats CAGrad and Nash-MTL, which compute and store one gradient per task. We borrow FAMO's idea of tracking loss progress cheaply, but we apply it to the data generator instead of the loss weights. Scalarization for Multi-Task and Multi-Domain Learning at Scale and In Defense of the Unitary Scalarization show that plain or tuned loss weighting often matches gradient surgery. The first also reports that gradient conflict rises late in training even as performance improves. We take this as a constraint: we do not propose a new gradient-surgery method, and we treat tuned scalarization as a serious competitor.

The visuo-tactile manipulation work motivates the occlusion focus and supplies the policy architecture. DECO adds a tactile adapter that enters a vision-action diffusion transformer only through cross-attention, which is attention where visual tokens query tactile tokens for relevant content. That adapter raised average success on DECO-50 from 72.25% (vision only) to 82.50%, with a 20-point gain on the most occluded tasks. Touch in the Wild reports 85% versus 25% for vision only on occluded test-tube collection. Multi-Modal Policy Consensus shows that naive feature concatenation collapses to 5% on an occluded picking task, compared with 65% for modular fusion. None of these works generate synthetic data, and none study how tasks compete. We reuse a DECO-style cross-attention adapter with a learnable gate from Gated Cross Attention. That gate mixes cross-attended context with the original features, and its value tells us when the policy is actually relying on touch. We use the gate as a diagnostic, not as the contribution.

The synthetic-data work supplies the generators and frames the cost problem. MobileH2R generates 100,000 scenes and adds a loss to the motion planner so the arm does not block the cameras. It transfers zero-shot at 80% on simple handovers, but it has no touch and is expensive to plan. Elastic Tactile Simulation (EIP) gives high-fidelity soft-sensor physics at heavy cost. MobileH2R works against occlusion by planning trajectories that avoid it. We take the opposite stance: occlusion during contact is unavoidable, so we concentrate tactile data there. I found no paper in the file that allocates simulation budget across tasks adaptively.

Research Program

The system has three parts.

The first is a multi-task visuo-tactile policy: a diffusion policy over rendered RGB-D views, proprioception, and simulated tactile flow maps, with touch entering through a gated cross-attention adapter. It is trained with ordinary summed imitation loss unless a baseline specifies otherwise.

The second is a generator that can do two things. It can produce full demonstrations with a scripted or sampling-based planner. It can also produce cheaper phase-targeted segments by restoring a cached simulator state shortly before contact, perturbing object pose and friction, and re-planning only the terminal approach. Each trajectory is labelled frame by frame as occluded or visible. The label comes from whether the rendered gripper mask covers the contact points in every camera, which the simulator knows exactly.

The third is the allocation controller. After round $k$ of training, it reads held-out imitation loss per task $t$ and phase $p$ (free-space or occluded-contact). It allocates the next round's budget $B$ as

$$ b_{t,p} \;=\; B\,\frac{\exp\!\big(\beta\,(\ell_t-\bar\ell)\big)\,\max(\rho_{t,p},\epsilon)}{\sum_{t',p'}\exp\!\big(\beta\,(\ell_{t'}-\bar\ell)\big)\,\max(\rho_{t',p'},\epsilon)}. $$

Here $\ell_t$ is task $t$'s current log held-out loss and $\bar\ell$ is its mean over tasks. The exponential term up-weights lagging tasks, as FAMO does. The temperature $\beta$ controls how strongly lagging tasks are favored. The quantity $\rho_{t,p}$ is the observed drop in log loss for that task and phase per simulation-second of data added in the previous round. It measures marginal return and stops the controller from feeding tasks that are stuck no matter how much data they get. The floor $\epsilon$ keeps every cell explored. The controller state is $2T$ scalars, with no per-task gradients, so its cost does not grow with model size.

Held-out imitation loss is a proxy for success. The program therefore measures how well per-task loss rank correlates with per-task rollout success at each round. If the correlation is poor, the signal is swapped for a small number of cheap rollouts per task, and that rollout cost is charged to the budget.

As a secondary analysis we log gradient cosine similarity between tasks, separately for the visual encoder, the tactile cross-attention adapter, and the action head. This tests whether interference in shared cross-attention layers predicts which tasks the controller starves or favors. The question is open in the file.

Baselines

The naive reference is uniform generation with uniform loss: the same number of full demonstrations per task until the budget is spent, with losses summed equally. This is how RVT- and PerAct-style multi-task policies are typically trained.

The strongest existing loss-side competitor is uniform generation plus FAMO loss weighting, and separately uniform generation plus loss weights tuned by population-based training (PBT), following the scalarization paper. Tuned scalarization beat PCGrad and CAGrad on DomainNet, 48.01% versus 42–43%, so it is the strongest learning-side method in the file. Both get the same data, the same network, the same number of gradient steps, and the same tuning budget, counted in total training runs, as our controller's $\beta$ and $\epsilon$ search. PBT's extra training runs are reported separately so readers can see that cost.

The strongest existing policy references come from the literature. On real data, DECO with its tactile adapter reports 82.50% average success on DECO-50. On RLBench, RVT reports 62.9% average success over 18 tasks, against PerAct's 49.4%. On MetaWorld MT10 the file gives no numeric FAMO success rate: no reported result found on Lacuna for FAMO's MT10 success value. For a simulated multi-task visuo-tactile suite there is likewise no reported result found on Lacuna. All numbers on that suite will therefore come from our own reruns.

Three ablations isolate the key ingredients. Task-only allocation removes the phase dimension: same controller, full demonstrations only. It tests whether targeting occluded windows matters, or merely giving hard tasks more data. Phase-only allocation splits the budget uniformly across tasks but targets occluded segments. Allocation without the marginal-return term $\rho$ reduces the controller to "feed whoever lags." Our controller beats the field only if, at equal total generation seconds, its average success exceeds the best of FAMO-on-uniform-data and PBT-on-uniform-data by more than seed-to-seed variation (five seeds). It must also exceed task-only allocation, and the gap must be concentrated in occluded-phase failures.

Why not just…?

Why not just reweight the losses with FAMO or tuned scalarization on a uniformly generated dataset? This is the real threat. Loss reweighting is free at generation time, and the scalarization paper shows that tuned weights are hard to beat. The argument against it is structural, not cited. Reweighting can only emphasize examples that already exist. If a task's occluded contact windows are a few frames per trajectory, upweighting that task also upweights its many free-space frames. Reweighting cannot create new grasp variations either. The controller-versus-FAMO-on-uniform-data comparison settles it directly. If they tie, the conclusion is that generation-side balancing adds nothing, and the paper reports that.

Why not just add cameras or virtual views instead of simulating touch? RVT re-renders point clouds into virtual views to see past the arm, and active vision moves the camera. Both still cannot see contact surfaces enclosed by fingers. The evidence is consistent: test-tube collection reaches 25% with vision versus 85% with touch in Touch in the Wild, and cable mounting reaches 0.15 versus 0.91–1.00 in LEMMo-Plan. We nonetheless include an RVT-style vision-only policy with matched generation budget, to measure what touch is worth per simulation-second.

Why not just use the cheap tactile simulator for everything, at a large uniform budget? At 1000 FPS the tactile physics may not dominate cost. Planning, rendering, and rejected trajectories may dominate instead, as MobileH2R's pipeline suggests. If so, allocation could matter less than raw volume. We report a full cost breakdown, and we sweep total budget. If uniform generation catches up once the budget is roughly doubled, the honest claim shrinks to a constant-factor saving. That is still useful, but it is modest.

Proposed Experiments

The main testbed is a new 12-task suite built on the differentiable penalty-contact tactile model. It covers that paper's peg insertion, box pushing, cap opening, and grasping, plus variants such as deep-bin picking, lid closing, and connector mating. External and wrist cameras are positioned so that the gripper occludes contact by construction. Success is measured on 100 fixed held-out initial states per task. We plot success against cumulative generation seconds, which is the headline figure. We also report success restricted to failures that occur inside occluded windows.

The decisive check is the comparison at matched budget between our controller, FAMO-on-uniform-data, and task-only allocation. A secondary check tests whether the cross-attention gate opens mainly in occluded frames. If it does not, the policy is not using touch where we claim it matters, and the phase story is wrong even if success improves.

The vision-only form of the principle is tested on RLBench's 18 tasks with RVT as the policy. Its scripted demonstration generator is held to a fixed total generation budget, and RVT's reported 62.9% serves as the external reference for the standard setup.

Real-world relevance is probed in two ways. First, we train the gated adapter on DECO-50 to confirm that the architecture is not weaker than DECO's reported 82.50%. Second, we deploy suite-trained insertion and grasping policies zero-shot on a GelSlim-equipped arm, comparing controller-allocated against uniform data at equal budget.