A generalist robot policy is one network trained on demonstrations from many different robots. Those robots come with different cameras. Some datasets have a wrist-mounted camera and a fixed third-person camera, some have only the third-person view, and some have cameras placed at angles no other dataset uses. The policy sees all of this as a flat list of image tokens, and the list itself reveals which dataset, and therefore which robot, it is looking at. We argue that this is the hidden reason generalist policies neglect wrist cameras. The problem is not only that wrist data is scarce. Camera composition is a free label for embodiment, and the policy can use it to pick up that robot's action habits without looking at the object. Octo reports this outcome without diagnosing its cause: wrist-camera performance was worse than expected, and only 27% of its training data contained wrist views.
We propose to remove this confound in the architecture. The policy no longer receives per-camera image tokens. It receives a fixed set of 3D point tokens anchored to the gripper. Each token gathers features from whichever cameras happen to see it, and a learned per-camera gate weighs each contribution. The token interface is then identical whether one camera, two, or five are present, so the camera set can no longer act as an embodiment fingerprint. Wrist evidence enters wherever the wrist camera actually sees the workspace, which is exactly the region that matters during contact. We pair the architecture with a counterfactual diagnostic of how much each camera actually drives the chosen action. We test the central prediction directly: a policy with a camera-invariant interface should shift its reliance toward the wrist view when the third-person view is occluded, and a flat-token policy trained on a camera-confounded mixture should not.
Core Claim
In cross-embodiment training, viewpoint imbalance is partly a shortcut problem rather than purely a data-scarcity problem. It follows the pattern documented by Shortcut Learning in Generalist Robot Policies, which shows that fragmented pooled datasets drive policies to latch onto camera angle, background, and similar task-irrelevant cues before they learn the task. Two ingredients should reduce this shortcut, and together they should do so more reliably than data-level fixes. The first is a gripper-anchored point-token interface that is invariant to camera composition. The second is gated per-camera aggregation. The claim is falsifiable. If a fair data-level fix makes flat-token policies equally robust, namely balanced resampling of camera compositions combined with random view dropout, then the architectural claim is wrong and we will report it as such.
How This Relates to Prior Work
Three works motivate the gap. Octo trains one transformer on 800k Open X-Embodiment trajectories. It uses block-wise masked attention so that cameras can be added or dropped, but it treats each camera as its own group of 2D patch tokens and observes wrist neglect. Shortcut Learning shows that inter-dataset disparity produces shortcuts and fixes them with data augmentation and "bridge" objects. It does not try architectural fixes. MV-UMI adds a third-person camera to handheld data collection and prevents collapse onto one view with random viewpoint dropout. It concatenates 2D tokens and works within a single embodiment, where camera composition carries no information about the robot.
Two lines of work supply the method. BEVFormer and Occupancy as Set of Points let 3D query points project into each camera and sample features with deformable cross-attention, where each query samples a few learned offsets around its projected location. BEVFormer notes that this sampling is somewhat tolerant of small calibration errors. We reuse this backward-sampling idea, with two changes. We move the queries from a vehicle-centric driving grid to the robot's gripper frame. We also make aggregation over cameras explicitly gated, following the gated cross-attention of Kim et al., which throttles cross-modal flow so that a dense stream cannot wash out a sparse one.
The closest competitor is RoboMM. Its UVFormer adapter already lifts multi-view images into a 3D voxel grid inside a cross-embodiment policy trained on RoboData, a merged pool of CALVIN, Meta-World, LIBERO, RoboCasa and RT-1. Our proposal differs in three ways. First, RoboMM's grid is world-centric, in a standardized "Right-Forward-Up" frame, whereas our points move with the gripper. Workspace scale therefore differs less across robots, and wrist-camera projections depend only on the fixed hand-eye mount. Second, RoboMM does not measure or target reliance on individual cameras. Third, RoboMM uses dense voxels, while we use a sparse point set sized to match the flat baseline's token budget.
Two works constrain the design. Embedding Morphology into Transformers found that soft, continuous attention biases were unstable to train compared with hard masks. Our gates are soft, so we bound them, initialize them open, and report training stability as an outcome rather than assuming it. 3DFlowAction argues that rigid coordinate frames break cross-embodiment transfer. A gripper-anchored frame is our partial answer to that objection, and the world-voxel comparison tests whether the frame choice matters.
The modality-bias literature supplies the measurement philosophy. LLaVA Steering quantifies imbalance with a Layer-wise Modality Attention Ratio (LMAR). Diagnosing Vision-and-Language Navigation shows that counterfactual ablations reveal "direction junkies" whose attention maps look grounded. We follow the second lesson: attention weight is not reliance. Our primary metric is therefore counterfactual, and LMAR-style attention ratios are secondary.
Research Program
The interface is a fixed set of $P$ query points laid on a lattice in the gripper frame, with a coarse outer shell for scene context. For point $p$ and each available camera $v$, the point is projected into that camera using known intrinsics and extrinsics together with forward kinematics. A deformable cross-attention step then samples image features $F_v$ around the projection. The point token is
$$z_p = \sum_{v \in \mathcal{V}} \frac{g_{p,v}\, m_{p,v}}{\sum_{u \in \mathcal{V}} g_{p,u}\, m_{p,u} + \epsilon}\, \mathrm{DeformAttn}\big(q_p, F_v, \pi_v(p)\big).$$
Here $\mathcal{V}$ is whatever set of cameras this robot has, $\pi_v(p)$ is the 2D projection of $p$ into camera $v$, and $m_{p,v}\in\{0,1\}$ marks whether the projection falls inside the image. The gate $g_{p,v}=\sigma(w^\top[q_p; \bar F_v(\pi_v(p))])$ is bounded in $(0,1)$ and computed from the query and the locally pooled feature $\bar F_v$. Finally, $\epsilon$ prevents division by zero for points no camera sees; such points receive a learned "unobserved" embedding. The normalization is the key design choice. Removing a camera changes how much evidence each point has, but it never changes the number, order, or type of tokens.
Language and proprioception tokens are added as in the flat baseline. The action head is an unchanged diffusion head, so every difference lies in the visual interface.
The diagnostic measures view reliance for camera $v$ as
$$R_v = \mathbb{E}_{o}\Big[ D\big(\pi(\cdot \mid o),\ \pi(\cdot \mid o_{\setminus v})\big) \Big],$$
where $o_{\setminus v}$ is the observation with camera $v$ replaced by its training-time "absent" treatment and $D$ is the mean squared distance between denoised action chunks drawn with fixed noise seeds. The target is not equal reliance. The target is reliance that tracks informativeness. We therefore report the occlusion-conditional shift: how much $R_{\text{wrist}}$ rises when the third-person view of the target object is artificially occluded. A policy that uses its cameras correctly should show a large shift. A policy using camera composition as an embodiment code should show almost none.
The controlled confound study sits at the center of the program. Using RoboData's simulated components, we build two training mixtures from the same episodes. In the confounded mixture, camera composition is tied to source suite. For example, the wrist view is kept for LIBERO and CALVIN episodes and removed for a matched fraction of other suites, mimicking the camera statistics Octo describes. In the decorrelated mixture, the same total number of wrist frames is spread uniformly across suites. Comparing each architecture across the two mixtures isolates how much of its wrist neglect comes from the confound rather than from wrist scarcity.
Baselines
The naive reference point is a flat-token policy in the style of Octo. Per-camera patch tokens are concatenated and fed to the same transformer and diffusion head, with no augmentation. It is trained on the same RoboData subset for the same number of steps. Every other method must beat it, and its confounded-versus-decorrelated gap is the quantity we expect our method to shrink.
The strongest data-level competitor is the flat-token policy trained with random viewpoint dropout, following MV-UMI, plus balanced resampling of camera compositions per suite. This is the answer a skeptical reviewer would give. We give it the same hyperparameter-search budget as our method, including the dropout rate, and train it on identical data. Our method beats it only if it wins on the held-out-composition and occlusion evaluations by more than the seed-to-seed spread across three seeds, while not losing in-distribution success.
The strongest existing architecture is RoboMM's world-frame voxel lifting. On LIBERO, RoboMM reports a "91.7%" success rate, and on CALVIN an average task sequence length rising "from 1.7 to 3.5" (RoboMM). We reimplement a UVFormer-style world-grid adapter at a matched token count on our backbone. We will not compare against the published number directly, because backbone and training data differ; the published figures serve only as a sanity range for our reimplementation.
For camera-viewpoint robustness, the reference is the view-synthesis wrapper AnyCamVLA, which reports "94.5%" average LIBERO success across unseen viewpoints versus "67.9%" for its base policy. We use it as an external reference on the unseen-viewpoint split. It solves a different problem, re-rendering one canonical view, and it cannot use a camera the base policy never trained with.
Two ablations isolate the key ingredients. The first keeps the point tokens but replaces the gated, normalized sum with a plain average over visible cameras. It tests whether gating matters beyond the invariant interface. The second keeps gating but uses world-frame points instead of gripper-frame points. It separates anchoring from aggregation.
Why not just…?
Why not just use viewpoint dropout and balanced sampling? This is the real threat. MV-UMI shows that dropout prevents collapse within one embodiment. Dropout alone, however, cannot break the confound. A dataset that never had a wrist camera still never shows one, so the policy can still read the embodiment off the camera set. Balanced resampling can break the confound only when the data needed to balance exists. In real pooled datasets it often does not, which is why we test it in simulation where we control composition. If this baseline matches our method on the confounded mixture's occlusion-conditional reliance shift, the architectural claim fails. The experiment is built to show that outcome if it is true.
Why not just wrap a frozen policy with novel view synthesis? AnyCamVLA handles camera moves well, but it maps everything into one canonical 2D view. A region hidden from that reference view cannot be recovered, and the wrapper adds a 171M-parameter model that itself needs calibration. Our occlusion split directly tests the case where only the wrist camera sees the target, a case the wrapper cannot address by construction.
Why not just use RoboMM's voxel grid? It already lifts views to 3D across embodiments and reports strong numbers. It is included as a matched baseline. The two methods make different predictions under the confounded mixture and under extrinsic perturbation. A world grid needs the base-to-camera transform for every view and has workspace extents that differ by robot, while gripper-anchored points need only the hand-eye mount for the wrist camera. If the world grid matches us on these splits, gripper anchoring is not what matters, and the ablation will say so.
Why not just steer visual tokens at inference, as MoReS does? LLaVA Steering rebalances vision against text in static question answering. Applied here, it would reweight camera tokens without removing the embodiment signal carried by which camera tokens are present. We include a steered flat-token variant only as a secondary check, because the cited evidence says nothing about action policies.
Proposed Experiments
Training uses the simulated RoboData components: LIBERO, CALVIN, Meta-World and RoboCasa. These provide exact camera parameters and heterogeneous camera sets, in both the confounded and decorrelated mixtures. Evaluation covers four splits. The first is in-distribution LIBERO suites and CALVIN sequence length. The second is a held-out camera composition, where an embodiment trained without a wrist camera is tested with one and vice versa. The third is target occlusion, where a rendered occluder blocks the third-person view of the object. The fourth is extrinsic perturbation of increasing magnitude, following AnyCamVLA's unseen-viewpoint protocol. We report success rate, CALVIN average length, per-camera reliance $R_v$, and the occlusion-conditional reliance shift, with LMAR-style attention ratios as a secondary diagnostic.
A real-robot check uses the dual-camera MV-UMI task suite. There, MV-UMI reports a "47% performance improvement over single-view baselines", and the operator-masking ablation drops to "0.10" relative performance. We test whether gated point tokens match dropout within one embodiment, where no confound exists and we predict no advantage. We also track gate trajectories during training, because of the instability Suzuki et al. observed with soft biases.
Three results decide the question. The first is whether our method's gap between the confounded and decorrelated mixtures is smaller than that of the flat-token and dropout baselines. The second is whether its occlusion-conditional wrist-reliance shift is large under the confounded mixture while the baselines' shifts are small. The third is whether the within-embodiment MV-UMI comparison shows no advantage, which is what separates a confound explanation from a generic "3D is better" explanation. If all three hold, camera-invariant interfaces belong in generalist policies. If the dropout baseline closes the first two gaps, the field should invest in data balancing instead.
References
- Octo: An Open-Source Generalist Robot Policy
- Shortcut Learning in Generalist Robot Policies: The Role of Dataset Diversity and Fragmentation
- MV-UMI: A Scalable Multi-View Interface for Cross-Embodiment Learning
- BEVFormer: Learning Bird's-Eye-View Representation from Multi-camera Images via Spatiotemporal Transformers
- Occupancy as Set of Points
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- RoboMM: All-in-One Multimodal Large Model for Robotic Manipulation
- Embedding Morphology into Transformers for Cross-Robot Policy Learning
- 3DFlowAction: Learning Cross-Embodiment Manipulation from 3D Flow World Model
- LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering
- Diagnosing Vision-and-Language Navigation: What Really Matters
- AnyCamVLA: Zero-Shot Camera Adaptation for Viewpoint Robust Vision-Language-Action Models