Robot manipulation policies that act in 3D face a resolution budget problem. A uniform voxel grid, meaning a regular 3D lattice of cells, fine enough for millimetre precision over a one-metre table is unaffordable. The strongest current answer is to zoom: score a few coarse candidate positions, then re-sample finer candidates inside a small box around the best one. That zoom schedule is set by hand today. An engineer chooses the workspace box and the box sizes at each level, much as a traffic engineer draws entry polygons on every intersection camera. We propose to let the policy size its own zoom boxes from its own uncertainty. When the coarse stage is confident, the next box shrinks and the fixed point budget buys fine resolution. When it is unsure, as it often is on real camera images after training in simulation, the box stays wide enough to contain the true target.
The claim that matters is about transfer. A fixed zoom schedule tuned in simulation fails in a specific way on real hardware. The coarse stage picks a slightly wrong centre under an appearance shift, and the next 16 cm box no longer contains the target, so no later refinement can recover. The failure is silent. We argue that domain randomization, which trains in simulation while varying textures, lighting, clutter and cameras so the real world looks like one more variation, does more than make features robust. It is also what makes the coarse stage's spread of scores a trustworthy signal for how large the next region must be. Letting that signal set the region size should turn silent real-world misses into a slightly coarser but still correct refinement.
Core Claim and Mechanism
Act3D (Act3D: Infinite Resolution Action Detection Transformer for Robotic Manipulation; also Act3D: 3D Feature Field Transformers) lifts multi-view image features into a continuous 3D feature field. It scores sampled "ghost points" (candidate end-effector positions) in three stages: a fixed 1 m workspace, then a 16 cm box, then a 4 cm box, each centred on the previous best point. Act3D supplies the method we reuse. Its fixed box sizes are the manual configuration we remove.
At each level $l$ we already have scores over $N$ candidate points. Let $\Sigma_l$ be the covariance of candidate positions, weighted by the softmax of their scores. We set the next box half-width as
$$r_{l+1} = \mathrm{clip}\big(\kappa \sqrt{\lambda_{\max}(\Sigma_l)},\ r_{\min},\ r_l\big),$$
where $\lambda_{\max}$ is the largest eigenvalue of $\Sigma_l$ and $\kappa$ is a single coverage multiplier. The box can never grow, and it cannot shrink below the physical resolution $r_{\min}$ that the task needs. Refinement stops when $r_{l+1}$ reaches $r_{\min}$. The number of levels is therefore also adaptive. A confident prediction reaches millimetre scale in few steps, and an unsure one spends its budget holding a wider box.
When the score distribution has several separated modes, for example two identical mugs, we keep the top two modes as separate boxes that share the same point budget. We add this only if single-box coverage is shown to fail on such scenes.
The constant $\kappa$ is chosen once, in simulation, on held-out randomizations. It is set so that the true target lies inside the next box in a chosen fraction of cases, say 95%. This is the one manually specified quantity left. It is a probability target, not a geometric length, so it does not need re-tuning per task or camera.
The initial workspace is not a learned object. It is the robot's reachable volume intersected with the bounding volume of observed depth points. Calling that "automatic ROI discovery" would overclaim, so we don't.
Where This Sits Relative to Prior Work
Three works motivate the gap. Coarse-to-Fine for Sim-to-Real reaches a hand-chosen "bottleneck pose" with pose estimation and ICP, then hands off to a domain-randomized eye-in-hand policy. It reaches 90–100% on 0.5 mm peg insertion against 10–15% for a monolithic policy. This shows that staging a local region is what makes precision possible, but its region is hand-built and has exactly two stages. Act3D automates the staging inside a learned model, but it was not stress-tested under aggressive randomization. PointRend refines segmentation masks only at uncertain points, and it is known to be unable to recover when the coarse stage misses. That is exactly the failure a fixed box has under a domain shift.
We borrow PointRend's principle that uncertainty should decide where to spend resolution. We apply it to the size of the region, not only to point placement, and in 3D action space instead of 2D masks.
Two works constrain the design. Robust Visual Sim-to-Real Transfer for Robotic Manipulation finds that realistic AmbientCG textures beat procedural ones and that excessive camera and colour randomization hurts. We therefore adopt its tuned randomization settings rather than maximal randomization. The tension between "diversity beats fidelity" in (CAD)$^2$RL and fidelity-first work such as Sim-to-Real: Learning Agile Locomotion For Quadruped Robots is left alone, because we vary appearance only and use the benchmark physics.
ManiSkill2 reports that point clouds beat RGB-D images. Coarse-to-Fine for Sim-to-Real reports that close-range depth is too noisy for sub-millimetre work. This bears directly on us, because Act3D's feature lifting depends on depth. We treat it as a hypothesis to measure: depth noise should inflate $\Sigma_l$ at the fine levels, and the adaptive rule should respond by stopping refinement early instead of committing to a wrong fine point.
A more distant motivation comes from traffic counting. Zero-VIRUS* replaced hand-drawn zones with continuous proximity and completeness fields, reaching an overall score of 0.9292 on AI City 2020 Track 1. The gain from the full set of fields over proximity alone was 0.9554 versus 0.8903. It still required hand-drawn route polylines, and Virtual Traffic Lanes still needs GUI-drawn polygons. The general lesson is that soft, data-driven spatial support beats hard drawn boundaries. We do not evaluate on traffic.
Research Program
We build the adaptive-radius rule on top of an Act3D reimplementation. The architecture, feature backbone and total ghost-point budget per decision stay fixed, so the only change is how region size is chosen.
We train on RLBench (RLBench) with appearance randomization following the tuned settings of Robust Visual Sim-to-Real Transfer: realistic textures, lighting, object colour and mild camera jitter. We construct held-out shift splits using unseen texture sets, lighting outside the training range, distractor objects and camera viewpoint changes.
The primary diagnostic is not success rate but region coverage: the fraction of decisions in which the ground-truth target lies inside every box the policy refines into. Alongside it we measure final position error and success. A fixed schedule should show coverage collapsing under shift while its error given coverage stays low. The adaptive rule should keep coverage near the $\kappa$ target at the cost of a somewhat larger error given coverage. The net effect on success is the open empirical question.
We then test physically. Following the Act3D real-robot protocol, we use a Franka Panda on a subset of tasks with zero real fine-tuning. We add one precision task, peg insertion at a few-millimetre clearance, where the resolution budget actually binds.
Baselines
The naive reference is the Act3D-style policy with its published fixed 1 m / 16 cm / 4 cm schedule, trained without randomization. It shows how far plain transfer falls. The strongest existing method is the same fixed-schedule Act3D trained with our identical randomization. Act3D reports 83% single-task success over 74 RLBench tasks and 65% multi-task success over 18 tasks. Its success drops 24% under novel viewpoints versus 77% for HiveFormer, and it beats PerAct by 22% with 3x less compute. These numbers were measured without aggressive randomization. We found no reported result on Lacuna for Act3D under held-out appearance shifts, or for its zero-shot sim-to-real success after simulation-only training. Its fixed schedule is exactly what our method replaces, which makes it the most relevant competitor. It gets the same data, randomization, training steps and point budget, and its checkpoint is selected on the same held-out simulated randomizations.
A uniform-voxel policy in the style of PerAct, trained with the same randomization, anchors the "just use a dense grid" end of the trade-off at matched memory.
For precision transfer, the reference is the two-stage bottleneck controller of Coarse-to-Fine for Sim-to-Real, with its reported 90–100% on 0.5 mm insertion. We reimplement it on our insertion task, hand-tuning the bottleneck pose, and count the engineer time this takes as part of the comparison.
The ablation that isolates our key ingredient trains the adaptive-radius policy without domain randomization. If region size from uncertainty helps only when randomization has made that uncertainty meaningful, this variant should lose coverage on real images even though it uses the same rule.
We count our method as beating the strongest baseline if it significantly raises success on the held-out shift splits and on the real robot across seeds, and does not lose more than a point or two on in-distribution RLBench. The coverage gap should explain the difference.
Why not just…?
Why not just tune the fixed box sizes larger, or add a fourth level? This is the most serious threat, since a wider fixed schedule also buys coverage. We sweep fixed schedules, both box sizes and number of levels, at matched point budget. We plot coverage against error given coverage on the held-out shifts, and ask whether the adaptive rule lies above that frontier. If a single fixed schedule tuned on simulated randomizations matches it on real transfer, the paper's claim is wrong and we will say so. The prediction is that it cannot, because the right size differs between confident and unconfident decisions within the same task.
Why not just keep the top few coarse candidates (beam search) instead of resizing one box? Beam search spends budget on alternatives rather than on spread. It helps with multimodal scenes, where two mugs are genuinely distinct candidates, but not with a single smeared prediction. We include a top-$k$ beam variant at matched budget. If it matches us, the contribution reduces to "hedge the coarse stage," and the multi-mode extension becomes the method.
Why not just use a hand-set bottleneck pose, as in Coarse-to-Fine for Sim-to-Real? It works, at 90–100% on 0.5 mm insertion in Coarse-to-Fine for Sim-to-Real, and on a single known task it may well beat us. Its cost is that the bottleneck must be re-engineered whenever the object's position range or geometry changes. Our test is multi-task RLBench plus physical tasks with randomized object placement, where one hand-set bottleneck per task has to be built. We report both success and setup effort.
Why not skip simulation and collect real demonstrations? The file's evidence argues against this under visual shift. Real-only policies fell to 5% on a textured tablecloth against 100% for the randomized policy in Robust Visual Sim-to-Real Transfer. Randomized simulation pre-training gave a 3.6x few-shot gain over real-only data in RoboTwin 2.0. We therefore do not re-run this comparison.
Decisive Checks
The first check is simulation-only. On held-out RLBench shifts, does fixed-schedule coverage drop while adaptive coverage stays near its 95% target, and does the no-randomization ablation lose that calibration? If adaptive coverage also collapses, uncertainty does not transfer and the idea fails cheaply, before any hardware time.
The second check is the fixed-schedule frontier described above.
The third check is the real Franka run. Zero-shot success on the transferred tasks must exceed the randomized fixed-schedule Act3D. On the insertion task, the adaptive policy should approach the hand-staged controller's success without a hand-set bottleneck pose. Logged region sizes should widen on real frames relative to simulation, which would confirm that the mechanism, not a side effect, drives any gain.
References
- Act3D: 3D Feature Field Transformers for Multi-Task Robotic Manipulation
- Coarse-to-Fine for Sim-to-Real: Sub-Millimetre Precision Across the Workspace
- PointRend: Image Segmentation As Rendering
- Robust Visual Sim-to-Real Transfer for Robotic Manipulation
- CAD2RL: Real Single-Image Flight Without a Single Real Image
- Sim-to-Real: Learning Agile Locomotion For Quadruped Robots
- ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
- Zero-VIRUS*: Zero-shot Vehicle Route Understanding System for Intelligent Transportation
- Towards Real-time Traffic Movement Count and Trajectory Reconstruction Using Virtual Traffic Lanes
- RLBench: The Robot Learning Benchmark & Learning Environment.
- RoboTwin 2.0: A Scalable Data Generator and Benchmark with Strong Domain Randomization for Robust Bimanual Robotic Manipulation