Robot policies that see the world as 3D point clouds learn from far fewer demonstrations than image-based ones, but the field disagrees about what they should do with those points. One camp compresses the whole scene into a single feature vector before deciding on an action. The other keeps a separate feature at every point. Both report strong results with a handful of demonstrations. We propose to resolve this disagreement with a controlled measurement and a small architectural fix. Our hypothesis is that a single scene vector is harmless when the geometry that matters for the task fills most of the observed scene, and becomes costly in demonstrations as that geometry shrinks relative to clutter. Examples of small task-critical geometry are a mug rim, a drawer handle, or a peg hole. The fix replaces "take the maximum of each channel over all points" with "keep the $k$ most task-relevant point features intact, with their 3D coordinates." This preserves fine local structure while keeping the fast inference of the compact design.
The stakes are practical. Demonstrations collected by a human remote-controlling the robot are the scarcest resource in robot learning, as the surveys and scaling studies in this area stress (A Survey on Robotics with Foundation Models, Scaling Up and Distilling Down). If the choice of how to summarize a point cloud silently multiplies the number of demonstrations needed in cluttered, fine-grained scenes, that is a cheap thing to fix. It is also a cheap thing to get wrong.
Core Claim
The apparent contradiction comes from two papers. 3D Diffusion Policy (DP3) encodes 512–1024 cropped, colorless points with a small per-point network and max-pooling into one compact vector. That vector conditions a diffusion action head, a generator that turns random noise into a trajectory by repeated denoising. DP3 reports a 74.4% average success rate over 72 simulated tasks with about 10 demonstrations each. SGRv2 argues the opposite. It says a single scene vector forces the policy to spend many demonstrations learning to ignore background. Its point-wise network predicts actions relative to each point, and it reaches 53.2% on 26 RLBench tasks with 5 demonstrations, against 40.4% for RVT.
We claim these results are not in conflict, because they were measured in different regimes. DP3's tasks mostly involve one object in a cropped workspace, where the critical geometry is a large share of the points. SGRv2's RLBench scenes are cluttered and multi-object.
The 3D shape-completion literature supplies the mechanism we expect. Learning Local Displacements for Point Cloud Completion shows that channel-wise max-pooling mixes features from different points into one summary, which erases thin structures. It proposes "neighbor-pooling," which keeps whole high-activation feature vectors instead. FullPart documents the analogous "resolution tax": small parts get only a few cells when everything shares one global grid.
Our testable prediction is quantitative. For a global-pooling policy, the number of demonstrations needed to reach a fixed success rate should grow as the fraction of points on task-critical geometry falls. For a policy that keeps $k$ intact local features, that number should stay roughly flat.
Research Program
The first component is a controlled benchmark that varies one quantity: the critical-geometry fraction $\rho$. We define $\rho$ as the share of observed points lying within a small radius of the contact location in the demonstrations. The radius will be fixed per task suite, and $\rho$ is computed automatically from demonstration end-effector poses at grasp and contact keyframes.
We vary $\rho$ in two ways, starting from tasks in the DP3 suite (MetaWorld, Adroit) and from ManiSkill2. The first is to add distractor objects drawn from the same asset pool. The second is to rescale the functional part relative to the whole object, for example shrinking a handle while keeping its body fixed. Each task gets roughly five $\rho$ levels. The point budget stays fixed, so a lower $\rho$ really does mean fewer points on what matters.
The second component is the encoder we propose, which we call keep-$k$ pooling. It is a drop-in replacement for DP3's max-pool.
- A shared per-point network produces a feature $f_i$ for each point $p_i$.
- A one-layer head scores each point's salience $s_i$.
- The $k$ highest-scoring points are kept whole as tokens $(f_i, p_i)$, with no channel mixing across points.
- The diffusion head attends to these $k$ tokens through cross-attention, a layer that lets each denoising step read from the tokens.
- A relative 3D position encoding is added, in the spirit of 3D Diffuser Actor.
- Gradients reach the salience scores through a soft top-$k$ during training.
- Optionally, SGRv2-style dense supervision can be added as an auxiliary loss. Every point predicts the action offset relative to itself.
Setting $k=1$ recovers something close to global pooling. Setting $k$ equal to the full point count recovers full point tokenization. The proposal therefore sits on a single axis we can sweep, which isolates whether intact local descriptors are the active ingredient.
The third component is analysis. For each encoder and each $\rho$ level, we fit success rate against the number of demonstrations (5, 10, 20, 40). From these curves we extract the demonstrations needed to reach 60% success. We also check whether the learned salience concentrates on the contact regions. This is measured as the fraction of kept tokens within the critical radius, with no part labels given, which mirrors SGRv2's observation that point weights find rims and handles.
Baselines
The naive reference point is DP3 itself, run unchanged with its own released hyperparameters on every $\rho$ level. Its reported 74.4% average over the 72-task suite (3D Diffusion Policy) is the number our reproduction must match at the unmodified $\rho$ level. Without that match, the comparison is not trustworthy. A second naive reference is DP3 with doubled point count, which tests whether simply sampling more points fixes small-part loss.
The strongest locality-preserving competitor on few-demonstration RLBench is SGRv2, which reports 53.2% average success with 5 demonstrations on 26 tasks (SGRv2). On our RLBench runs we will report against that figure and against its cited RVT figure of 40.4%. SGRv2 is a keyframe, deterministic behavioral-cloning policy, so on the continuous-control suites we will run its point-wise U-Net encoder under the same diffusion head instead of using its whole system.
The strongest dense-token diffusion competitor is 3D Diffuser Actor. It reports 81.3% on multi-view RLBench in its full-data setting (3D Diffuser Actor). We are aware of no figure for it in the 5-demonstration regime, so we will run it ourselves at 5 and 10 demonstrations.
All methods share the same demonstrations, cameras, point budget, action head where applicable, number of training steps, and a hyperparameter sweep of equal size, about 12 configurations. We report success with three seeds and 100 evaluation episodes per checkpoint.
Beating DP3 means a statistically significant gain at low $\rho$ with no loss at high $\rho$, at matched inference rate. DP3 reports 25.3 FPS. Beating the token baselines means matching their success at low $\rho$ while running at least twice as fast.
The key ablation sweeps $k$ from 1 to all points. It also replaces learned salience with random selection, and with selection by distance to the gripper.
Why not just…?
Why not just crop the point cloud tighter around the object? This is the most serious threat, because DP3 already relies on workspace cropping. If a hand-designed crop around the target recovers all of keep-$k$'s gains, then the real contribution is selection, not intact features. We test this directly. An oracle crop uses ground-truth contact locations, which is an upper bound for any cropping scheme. A gripper-proximity crop is the practical version. If the oracle crop plus max-pool matches keep-$k$, we will report that as the finding. The open question then becomes learning the crop, and the learned salience head is exactly such a learner. If keep-$k$ still wins under the oracle crop, that isolates channel mixing across points as the harm.
Why not just tokenize every point, as 3D Diffuser Actor does? It may well match keep-$k$ on success. However, it denoises over many multi-view tokens and runs around 10 Hz, using frozen 2D CLIP features rather than learned geometric ones. The $k$ sweep shows where on the cost–accuracy curve full tokenization sits. If it dominates even at matched compute, keep-$k$ has no reason to exist, and the curve will show this.
Why not just augment heavily and keep the global vector? Rotations and jitter make the vector robust. They do not restore descriptors that max-pooling has already mixed together, which is the argument of Learning Local Displacements. We will nonetheless include an augmented-DP3 arm, because that argument comes from shape completion and has not been tested on policies.
Why not just scale a 2D vision-language-action model on video? MotoVLA's ablation found 3D point tracking beat 2D pixel tracking by 12.5% in real tests (Generalist Robot Manipulation beyond Action Labeled Data). Our question also concerns the regime of tens of demonstrations, where such models are not designed to operate. We treat this alternative as out of scope rather than refuted.
Decisive Checks
The hypothesis lives or dies on one plot: demonstrations-to-60%-success against $\rho$, with one curve per encoder. The hypothesis is supported if DP3's curve rises steeply as $\rho$ falls while keep-$k$'s stays flat, and the random-selection ablation sits near DP3's curve. It is falsified if all curves are parallel, meaning global pooling costs a constant factor unrelated to clutter or part size, or if the oracle crop closes the gap.
A secondary check runs on a Franka arm. We test two fine-part tasks at 10 and 40 demonstrations with and without added distractors, comparing DP3 and keep-$k$. Success is reported alongside control rate, to confirm the effect survives real depth noise.
References
- A survey on robotics with foundation models: toward embodied ai
- Scaling Up and Distilling Down: Language-Guided Robot Skill Acquisition
- 3D Diffusion Policy: Generalizable Visuomotor Policy Learning via Simple 3D Representations
- Leveraging Locality to Boost Sample Efficiency in Robotic Manipulation
- Learning Local Displacements for Point Cloud Completion
- FullPart: Generating each 3D Part at Full Resolution
- 3D Diffuser Actor: Policy Diffusion with 3D Scene Representations
- Generalist Robot Manipulation beyond Action Labeled Data