Robot policies trained by imitation generalize to new rooms mainly when they have seen many different rooms. More demonstrations in the same room help very little. That diversity is expensive, and the evidence suggests most of it is spent teaching the camera-image part of the policy to ignore table colors, lighting and wall textures. Depth geometry, the 3D shape and position of things, barely changes when a room is repainted. This note proposes a policy that splits perception along that line. A slow pathway lifts rich image features from a pretrained 2D vision model (DINOv2) onto 3D points only once or twice per second, building a semantic map of what is where. A fast pathway sees only coarse depth geometry at control rate and tracks how things move. Semantics reach the action through a gated, distance-local lookup from fast geometric points into the slow semantic map, never through shared attention.
The claim is about data efficiency, not only accuracy. If appearance variation is confined to a stream that updates rarely and can only modulate geometry, then a policy should reach a target success rate in unseen environments with markedly fewer training environments than a policy that fuses color and depth at every frame. Concretely, the power law that relates out-of-distribution error to environment count should get steeper. The same split should also let the policy handle targets that move mid-reach, which static 3D policies cannot do.
Where the idea comes from
Three findings motivate the design. First, Data Scaling Laws in Imitation Learning for Robotic Manipulation showed that the zero-shot gap shrinks as a power law in the number of distinct environments and objects. It also showed that demonstrations per environment saturate around 50, and that scaling the visual encoder helps while scaling the action head does not. Perception is therefore what diversity is buying. Second, THE COLOSSEUM found that 3D voxel policies such as PerAct and RVT are robust to camera shifts but still lose 30–50% under lighting, texture and distractor changes, because their 2D feature extractors break. Third, SPOC trained an RGB-only agent over 200,000 procedural houses and reached 49.9%, which rose to 85% with ground-truth detection. Even huge diversity leaves a perception ceiling. Read together, these say that appearance is where diversity is consumed and geometry is comparatively cheap. No existing architecture exploits that split.
The fusion and temporal literatures supply the parts and also constrain the design. SlowFast Networks established that a high-resolution, low-frame-rate pathway paired with a low-capacity, high-frame-rate pathway captures appearance and motion at about 20% extra cost. Restricting the fast pathway's spatial capacity also acts as a regularizer against memorizing static detail, as Hao Li et al. also report. SlowFast-LLaVA-1.5 carried the idea to token budgets in video language models. Both operate entirely in 2D image space. We move the asymmetry to the interface where 2D features are lifted into 3D.
Two results rule out naive designs. BridgeVLA showed that inserting native 3D convolutions into a pretrained 2D model dropped RLBench success from 88.2% to 56.2%, so we keep the 2D encoder frozen and untouched. BrainWAM found that when clean semantic tokens and noisy dynamic tokens share one attention space, the action head over-attends to semantics, a failure it calls the "semantic shortcut." Our fusion is therefore structurally one-directional: geometry queries semantics, and the action head reads only geometric tokens. The gating mechanism itself comes from 3D-Mix for VLA, which found that semantic-conditioned adaptive gating beats concatenation and unconditioned cross-attention across nine fusion schemes. Any3D-VLA applies a related idea as a gated residual of 3D features onto DINOv2 features. We invert the direction, putting a gated semantic residual onto geometric tokens, and add time.
The closest prior systems each lack one piece. Cortical Policy already splits a static 3D "what" stream from a dynamic wrist-camera "how" stream. Its dynamic stream, however, is an RGB gaze tracker of the gripper trained on 3,600 narrow simulated clips, and it reaches only 24% on unseen tasks. Our fast stream sees only geometry, tracks the scene rather than the gripper, and is studied as a function of environment count. PointMapPolicy encodes RGB and structured point maps (an image-shaped grid of XYZ coordinates) and runs a sequence model over them at one rate. It is our strongest single-rate competitor, but it fuses appearance and geometry at every step and does not vary environment diversity.
Core Claim
Let $N$ be the number of distinct training environments, and let $G(N)$ be the success gap between a policy and its in-distribution ceiling on held-out environments. Following Lin et al., we model
$$G(N) \approx a\,N^{-\alpha},$$
where $a$ is a scale constant and $\alpha$ is the scaling exponent, fit by log-log regression as in that paper. Our hypothesis is that slow-semantic/fast-geometric decoupling raises $\alpha$ relative to single-rate RGB-D fusion at matched compute and tokens. A larger $\alpha$ means each added environment buys more generalization. We further predict that the improvement concentrates on appearance perturbation axes (texture, lighting, distractor color) and not on geometric ones (object size, camera pose). If $\alpha$ does not change, the idea reduces to an efficiency trick and we will say so.
Research Program
The policy has three parts. The slow pathway runs a frozen DINOv2 ViT-L on full-resolution RGB at a low rate $r_s$ (default 2 Hz). It projects each depth point into the image with the standard camera projection, samples the feature at that pixel, and voxel-pools the result into a few hundred 3D semantic tokens with metric coordinates. Between updates these tokens are held fixed, or optionally carried forward by rigid motion estimated from the fast stream. When something occludes an object, its semantics therefore persist from the last unoccluded view instead of projecting onto the occluder, which is one of the calibration and occlusion failures noted for point-wise early fusion in PointFusion and MVX-Net.
The fast pathway runs at control rate $r_f$ (default 10–20 Hz) on a low-resolution point map with no color. A small encoder with about one eighth of the slow channel width, following SlowFast's capacity ratio, processes it, and a short temporal window lets it represent velocity.
Fusion lets each fast geometric token gather the $k$ nearest slow semantic tokens by metric distance. It forms a semantic summary, and then adds it through a gate conditioned on a global semantic context vector, as in 3D-Mix. A diffusion action head, a generator that produces actions by iteratively denoising, reads only the fused geometric tokens. We optionally warm-start it with the consistency predictor from STEP (code) to keep the loop under 100 ms. This is an engineering choice and not part of the claim.
The scientific core is a scaling study. For each architecture we train on nested subsets of $N \in \{1, 4, 16, 64, 128\}$ kitchen environments with a fixed 50 demonstrations per environment-task pair, which matches Lin et al.'s saturation point. We then evaluate on disjoint held-out environments and fit $\alpha$ with bootstrap confidence intervals over seeds and environment draws.
Baselines
The naive reference is a 2D Diffusion Policy on RGB alone with the same frozen DINOv2 encoder and frame stacking, trained on the same environment subsets. It shows how much any 3D input helps under each $N$.
The strongest existing method on our main benchmark is PointMapPolicy, which reports 49.12% success on RoboCasa (PointMapPolicy), nearly 20% above DP3 and 3DA. It is the most direct competitor because it already combines RGB, structured 3D geometry and temporal modeling, at a single rate. We will run its public configuration on our environment subsets with the same demonstrations, the same frozen encoder where its design allows, the same action-head family and the same GPU-hours. Each method gets an equal hyperparameter search budget of the same number of configurations tuned on held-in validation environments. Beating it means a higher fitted $\alpha$ with non-overlapping confidence intervals, plus higher held-out success at $N \ge 16$. Matching its in-distribution score is acceptable.
On robustness we compare against Cortical Policy, which reports outperforming RVT-2 by 9.4% on COLOSSEUM stress tests and 80% on real moving targets. We also compare against BridgeVLA, which reports raising COLOSSEUM success from 56.7% to 64.0%. Both are trained on the RLBench-style COLOSSEUM training split with matched demonstrations. Beating them means exceeding 64.0% on COLOSSEUM while also handling moving targets, which BridgeVLA's single-step design cannot.
Two ablations isolate the key ingredient. The first is a single-rate twin: the identical architecture with the slow pathway run at control rate and its resolution reduced until token count and FLOPs match ours. This separates "asymmetric rates" from "more compute." The second is a shared-attention twin, in which slow and fast tokens enter one attention block that the action head reads directly. This tests whether the one-directional, distance-local gating matters, given BrainWAM's semantic-shortcut finding. We also include a geometry-only model in the style of 3D Diffusion Policy, which reports 85% on four real tasks with 40 demonstrations.
Why not just…?
Why not just drop color entirely and use a point-cloud policy like DP3? This is the real threat. If geometry is appearance-invariant, a geometry-only policy may already scale steeply with $N$. Its weakness is identity: it cannot tell a red mug from a blue one or a target from a same-shaped distractor. To settle it, we include task variants with color- and category-conditioned goals and shape-matched distractors. If DP3 matches us on the remaining tasks, the honest conclusion is that semantics are needed only for disambiguation, and that our method's value is getting that disambiguation without giving up DP3's scaling slope.
Why not just fuse RGB and depth at every frame, as PointMapPolicy does, and collect more environments? This is the comparison the single-rate twin answers directly. The prediction is not that single-rate fusion fails, but that it needs more environments for the same held-out success because its appearance features enter the action at full rate and full weight. If the fitted exponents are statistically indistinguishable, the core claim is false.
Why not just scale a 2D video policy across many more environments, as SPOC did? SPOC's own ablation is the counter-evidence: 200,000 houses left a gap from 49.9% to 85% that ground-truth perception closed. COLOSSEUM likewise shows 2D models (R3M, MVP) degrading sharply under appearance shifts. The RGB-only baseline in our scaling sweep measures this directly. If it shows the same exponent as ours, diversity alone suffices.
Why not just randomize textures and lighting in one simulated scene? Lin et al. found that within-setup variation saturates while variety of physical layouts and objects drives the power law, and 3D-MVP found that simulator-only pretraining did not survive COLOSSEUM perturbations. We add a heavy-randomization arm at $N=1$ for every architecture, so that we can report how many real layouts one randomized scene is worth.
Proposed Experiments
The main experiment is the RoboCasa scaling sweep described above. Held-out success is measured over unseen kitchens, and $\alpha$ is reported per architecture.
The second experiment breaks robustness down by perturbation axis on COLOSSEUM. Our prediction is specific: the decoupled model's advantage over the single-rate twin should be largest on texture, lighting and distractor-appearance axes and near zero on object-size and physical-property axes. A uniform gain across axes would suggest the benefit comes from capacity rather than from separating appearance from geometry.
The third experiment is a dynamic variant of the COLOSSEUM tasks in which the target is displaced mid-reach, mirroring Cortical Policy's moving-target protocol. We sweep $r_s$ from 0.5 to 10 Hz to find where stale semantics start to hurt.
The fourth is a diagnostic probe. At test time we swap the texture of every surface between slow updates, and separately shuffle slow semantic tokens across objects. Under the first, a model that has truly separated appearance from geometry should keep its motion tracking; under the second, it should fail only on identity-dependent tasks. We will log gate values to check that semantics are used rather than bypassed, and the reverse of BrainWAM's shortcut, where semantics are ignored, would also show up here.
The decisive check is the first experiment. A steeper fitted exponent at matched tokens and compute, with confidence intervals that separate from the single-rate twin and PointMapPolicy, supports the claim. Anything less means asymmetric lifting is at most a latency optimization.
References
- Data Scaling Laws in Imitation Learning for Robotic Manipulation
- THE COLOSSEUM: A Benchmark for Evaluating Generalization for Robotic Manipulation
- SPOC: Imitating Shortest Paths in Simulation Enables Effective Navigation and Manipulation in the Real World
- SlowFast Networks for Video Recognition
- A Novel Spatio-Temporal 3D Convolutional Encoder-Decoder Network for Dynamic Saliency Prediction
- SlowFast-LLaVA-1.5: A Family of Token-Efficient Video Large Language Models for Long-Form Video Understanding
- BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models
- BrainWAM: Action-Space Coordination of Semantic Priors and Predictive Dynamics for Autonomous Driving
- 3D-Mix for VLA: A Plug-and-Play Module for Integrating VGGT-based 3D Information into Vision-Language-Action Models
- Any3D-VLA: Enhancing VLA Robustness via Diverse Point Clouds
- Cortical Policy: A Dual-Stream View Transformer for Robotic Manipulation
- PointMapPolicy: Structured Point Cloud Processing for Multi-Modal Imitation Learning
- PointFusion: Deep Sensor Fusion for 3D Bounding Box Estimation
- MVX-Net: Multimodal VoxelNet for 3D Object Detection
- STEP: Warm-Started Visuomotor Policies with Spatiotemporal Consistency Prediction
- 3D Diffusion Policy
- 3D-MVP: 3D Multiview Pretraining for Robotic Manipulation