Robots that see the world as 3D point clouds (sets of surface points measured by a depth camera) transfer from simulation to reality better than robots that see images, because geometry does not change with lighting or texture. They still lose a lot on the way over. In the ManiSkill2 benchmark, a point-cloud cube-picking policy succeeds 91% of the time in simulation and 60% on a real robot. We argue that a large share of this loss has a specific cause. Simulated depth is complete and evenly spaced. Real depth cameras leave structured holes: they drop points at object silhouettes, on surfaces seen at grazing angles, and on shiny or dark materials. Graph-based point encoders build each point's features from its nearest neighbors, so these holes rewire the neighbor graph exactly at the edges and contact surfaces that a manipulation policy cares about.
The proposal is to train such policies against an adversary that removes points. The adversary is pushed in two directions at once. It is rewarded for changing the policy's actions, and it is penalized whenever its dropout patterns stop looking like real sensor holes. We embed this adversary in a hybrid imitation–reinforcement learning loop. A policy is first cloned from cheap scripted demonstrations in simulation, then improved by trial and error. The second stage matters because it is where the policy reaches states the demonstrations never covered, and so where the adversary can find failures that demonstration data never shows. We also propose a direct diagnostic, the fraction of each point's graph neighbors that survives real-sensor dropout. If our account is right, this number should predict real-robot success before any robot time is spent.
Core Claim
For graph-based point-cloud policies trained in simulation, the part of the sim-to-real gap that matters for control is concentrated in neighborhood rewiring caused by structured point dropout. A dropout adversary that is both policy-aware and constrained toward realistic dropout, applied during the reinforcement learning phase of a hybrid pipeline, should close more of that gap than random dropout or jitter. It should also do better than a task-agnostic learned dropout model, and it needs no digital twin or real action data. The hybrid setting is what makes the policy-aware part possible: the adversary needs on-policy states and a policy whose actions it can push around.
What the Literature Gives and Leaves Open
Graph point encoders supply the mechanism we probe. Dynamic Graph CNN introduced edge convolution. Each point's feature is computed from its own position and its offsets to its $k$ nearest neighbors, and in deeper layers the neighbor graph is rebuilt in the learned feature space. The paper reports 92.9% on ModelNet40 and robustness when half the points are removed at random. Random removal is not what real cameras do, and the research file itself flags feature-space graphs as prone to spurious edges under real noise. That is the failure we target. Liang and Boularias use DGCNN for manipulation from real demonstrations, but they only predict open-loop key-poses. Their work has no reinforcement learning and no simulation, so it cannot tell us how graph encoders behave across the sim-to-real boundary.
Two perception papers supply the method we adapt. Adaptive Spatial Masking learns differentiable, contextual point dropout for LiDAR segmentation, with a discriminator that keeps the dropout realistic. It lifts SynLiDAR→SemKITTI from 15.2% to 22.8% mIoU, against 20.0% for CoSMix. It also shows that uniform random masking does not capture real noise, which motivates our rejection of jitter-style augmentation. SIRA-PCR constrains learned resampling to stay inside the local hull of neighboring points, so that noise is realistic without bending the geometry. Both papers optimize realism only, for a fixed perception task. Neither asks which realistic corruptions change a controller's behavior, and that question is the new ingredient here.
The hybrid learning machinery comes from Imitation Bootstrapped Reinforcement Learning (IBRL). IBRL keeps a frozen imitation policy as a second action proposer, and the critic picks whichever proposal it values more. It reaches 100% on a real drawer task in 10,000 steps, but it works from images and stays within a single domain. ReAgent shows that cloning followed by PPO works on point clouds, using a global PointNet-style encoder and a registration task. On the sim-to-real side, DexPoint shows that point-cloud policies trained from scratch with oracle contact rewards can transfer: 87% on real bottles and 83% on cans. It does no dropout modeling and uses no demonstrations. RialTo closes the gap by building a digital twin of each real scene and robustifying in it. It reaches 90% versus 10% for imitation on Book-on-Shelf, at the cost of per-scene scanning and real demonstrations. Our method needs neither. It needs only a few hundred unlabeled real depth frames of the workspace.
Research Program
The pipeline has three learned parts.
The first is the policy: a point-cloud actor with an edge-convolution encoder. The critic is asymmetric. It receives the clean simulated cloud plus privileged object state, so the adversary perturbs only what the deployed actor sees and does not destabilize value learning.
The second is the dropout model. It assigns each point a drop probability from physically meaningful inputs: depth, the angle between the surface normal and the camera ray, distance to depth discontinuities, and a learned per-object reflectivity code. Binary masks are sampled with the Gumbel-Softmax relaxation, a trick for passing gradients through discrete choices, following ASM.
The third is a local-patch discriminator. It compares masked simulated patches with real ones, using patch statistics of density and holes rather than whole scenes, so that it cannot tell the domains apart by object content.
Write $x$ for a simulated cloud, $m$ for a binary keep-mask, $\pi$ for the actor, $D$ for the discriminator's estimate that a patch is real, and $\rho$ for a cap on the fraction of points dropped. The dropout model then maximizes
$$\mathcal{L}_{\text{adv}}(m) = \mathbb{E}_x\left[\, \lVert \pi(x) - \pi(m \odot x) \rVert^2 \,\right] + \lambda\, \mathbb{E}_x\left[\log D(m \odot x)\right], \quad \text{s.t. } \tfrac{1}{n}\textstyle\sum_i (1-m_i) \le \rho,$$
where $n$ is the number of points and $\lambda$ trades policy damage against realism. Setting $\lambda \to \infty$ recovers ASM-style realism-only masking, which is our key ablation. The actor is then trained on masked clouds, with alternating updates between the two.
Training proceeds in two stages, following IBRL. Behavior cloning on ManiSkill2's scripted simulated demonstrations produces the frozen imitation proposer, which is itself trained on masks from the dropout model. Off-policy reinforcement learning with the IBRL selection rule then continues while the adversary is active.
The diagnostic is the neighborhood survival rate. Let $N_\ell(i;x)$ be point $i$'s neighbor set at encoder layer $\ell$. The survival rate at layer $\ell$ is
$$S_\ell = \mathbb{E}_i\, \frac{|N_\ell(i;x) \cap N_\ell(i;x')|}{k},$$
the average fraction of neighbors that survive. Here $x$ is a simulated render of a scene and $x'$ is a real scan of the same physical arrangement, both restricted to points present in both. We compute $S_\ell$ for every trained variant on a small set of real scenes whose object poses are recovered by registration.
Baselines
The naive reference point is a point-cloud policy trained in clean simulation and deployed as is. Its anchor is ManiSkill2's reported PickCube result: 91% success in simulation and 60% on the real robot without fine-tuning (ManiSkill2). We rerun it on our hardware so that all comparisons share a camera and robot.
The strongest existing recipe for this question is IBRL-style hybrid training on point clouds with conventional augmentation, meaning random point dropout plus Gaussian jitter. It combines the best sample-efficiency method in the file (IBRL) with the DexPoint-style claim that geometry alone transfers. It is also what a practitioner would build today. We run it with the identical encoder, demonstrations, simulator steps, and hyperparameter search budget, and we tune its dropout rate and jitter scale over the same number of trials we spend tuning $\lambda$ and $\rho$. No reported result was found on Lacuna for this exact combination on ManiSkill2 tasks, so we establish it ourselves.
The ablation that isolates our key ingredient is realism-only learned masking: the same dropout model and discriminator with the policy-damage term removed. This is ASM transplanted to control. Its perception-side counterpart reported 22.8% mIoU versus 15.2% source-only on SemKITTI (ASM), but no manipulation result was found on Lacuna. We run a second ablation with the realism term removed ($\lambda = 0$). We expect this adversary to discover unrealistic, destructive masks and hurt real performance, which would show that realism is a necessary constraint rather than decoration. A third ablation is cloning only, with our masking applied and no reinforcement learning stage. It tests whether on-policy adversarial states matter.
All methods are crossed with three encoders. The first is a dynamic feature-space graph (DGCNN). The second is a static graph built once in 3D space. The third is a PointNet-style global encoder.
Beating the strongest baseline means three things. Our method must have higher real success on every task, with the PickCube margin outside the 95% binomial interval at 50 trials per method. Its sim-to-real drop must be smaller. And the gain must not vanish when we compare against realism-only masking. If realism-only masking matches us, the policy-aware term is unnecessary, and we would report that.
Why not just…?
Why not just add random point dropout and Gaussian jitter? ASM's ablations found that uniform random masking does not reproduce real dropout. Real holes are spatially correlated with angle, depth edges, and material (ASM). Random dropout mostly thins interiors, and interiors barely change a neighbor graph, whereas real holes cluster at silhouettes where grasps happen. That is the argument, not yet the proof. The strongest baseline above is exactly this alternative with a matched tuning budget. The survival-rate diagnostic will show whether random dropout already makes neighborhoods stable under real scans. If it does, and real success matches, our method is not needed.
Why not just drop the dynamic graph and use a static 3D graph or PointNet? This is the most serious threat. A static graph built in 3D cannot form spurious feature-space edges, and PointNet has no neighborhoods to rewire. Against that, PointNet-style encoders lose local surface structure: ManiSkill reports only 0.11–0.12 test success for cloning with PointNet plus Transformer (ManiSkill). R-PCR needed recurrence to stabilize ReAgent's global features (R-PCR). Even a static 3D graph is rewired when real holes remove the points at an object's edge. We do not argue this away. The encoder is a crossed factor in every experiment. If a static graph with plain augmentation matches our full method, the honest conclusion is that dynamic graphs are the problem, and our adversary is a fix for a self-inflicted wound. That finding would still be useful, and it would be visible in the encoder-by-augmentation results.
Why not just freeze the encoder after imitation so reinforcement learning cannot drift it? Freezing removes the representation drift that the research file flags for imitation-to-RL handoffs. It does nothing for sensor dropout at deployment, because a frozen encoder trained on clean simulated clouds is still rewired by real holes. We include a frozen-encoder variant of the strongest baseline. Our prediction is that it has better simulated stability and the same real-world drop.
Why not just build a digital twin, as RialTo does? RialTo's numbers are strong (90% versus 10% on Book-on-Shelf, RialTo). But it requires scanning and articulating each scene, plus real demonstrations, and its authors note that depth fails on thin and specular objects. A twin built from a real scan inherits the scan's holes as geometry; it does not model them as sensor behavior. Our method is complementary and much cheaper. We do not claim to beat RialTo on its suite and will not report a head-to-head we have not run.
Proposed Experiments
In simulation we use ManiSkill2 point-cloud tasks with the benchmark's demonstrations: PickCube, StackCube, PegInsertionSide, and OpenCabinetDrawer. We report success as a function of simulator steps for every method-by-encoder cell over five seeds. This checks that the adversary does not slow reinforcement learning to the point of erasing IBRL's sample-efficiency advantage. For real evaluation we reproduce PickCube, StackCube, and a drawer-opening task on a tabletop arm with a structured-light depth camera, at 50 trials per method per task. We add a stress condition with dark and glossy objects, where dropout is worst. The only real data used for training is several hundred unlabeled depth frames of the workspace, which feed the discriminator.
The decisive checks are three.
The first is whether the policy-aware term matters. Our method must beat realism-only masking on real success, most clearly in the glossy-object condition. If it does not, the core claim fails.
The second is whether neighborhood survival explains transfer. Across all trained variants, the edge-convolution layers' $S_\ell$ on paired sim and real scenes should rank-correlate with real success more strongly than simulated success does. If survival rate does not predict real performance, rewiring is not the mechanism, and the gains, if any, come from somewhere else.
The third is whether the hybrid stage carries its weight. The cloning-only variant with our masking should transfer worse than the full pipeline. This would show that adversarial pressure on states reached by reinforcement learning, rather than on demonstration states, is where the robustness comes from.
A cheap offline proxy, in the spirit of the cube-localization proxy in Robust Visual Sim-to-Real Transfer, lets us tune $\lambda$ and $\rho$ without robot time. We measure how much the actor's actions change between real scans and simulated renders of the same poses, and we select hyperparameters on that alone before any real rollout.
References
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- Dynamic Graph CNN for Learning on Point Clouds
- Learning Category-Level Manipulation Tasks from Point Clouds with Dynamic Graph CNNs
- Adversarially Masking Synthetic to Mimic Real: Adaptive Noise Injection for Point Cloud Segmentation Adaptation
- SIRA-PCR: Sim-to-Real Adaptation for 3D Point Cloud Registration
- Imitation Bootstrapped Reinforcement Learning
- ReAgent: Point Cloud Registration Using Imitation and Reinforcement Learning
- DexPoint: Generalizable Point Cloud Reinforcement Learning for Sim-to-Real Dexterous Manipulation
- Reconciling Reality through Simulation: A Real-To-Sim-to-Real Approach for Robust Manipulation
- ManiSkill: Generalizable Manipulation Skill Benchmark with Large-Scale Demonstrations
- R-PCR: Recurrent Point Cloud Registration Using High-Order Markov Decision
- Robust Visual Sim-to-Real Transfer for Robotic Manipulation