A robot that picks objects out of a cluttered bin usually runs a learned grasp detector once, executes the highest-scoring grasp, and, if that grasp fails, tries the next one. It learns almost nothing from its own failures within the session. Bayesian optimization is the textbook fix. It is a method that keeps a probabilistic model of which actions succeed and chooses the next trial to be maximally informative. Two costs have kept it out of fast grasping loops. The first is that choosing the next point means solving an inner optimization over an "acquisition function" (the score of how useful a trial would be). On the curved space of 3D gripper poses this inner problem is slow: roughly 1.5 s per iteration even for small orientation problems in Bayesian Optimization Meets Riemannian Manifolds in Robot Learning, and several seconds for Entropy Search in Automatic LQR Tuning Based on Gaussian Process Global Optimization. The second is that the local gradient-based pose refinement used by modern 6-DOF detectors is fragile to its step size.
We propose to remove both costs with one mechanism. Take the detector's proposals as starting points. Refine each by a few truncated proximal steps on the pose space. Such a step is a gradient step that is automatically shortened so it never jumps past the point where a linear model of the loss predicts it reaches zero. The quantity being refined is an optimistic success score that combines the detector with a small Gaussian-process correction learned from this session's physical outcomes. The refined poses form a small finite portfolio, and choosing among them is trivial. Acquisition optimization and grasp refinement therefore become the same computation. It costs a handful of network gradient evaluations, and its behavior barely depends on the step size.
Core Claim
The claim is that, for full 6-DOF grasping in clutter, where the gripper pose has three translations and three rotations and is not restricted to top-down approaches, a truncated-proximal refinement of an optimistic score does three things at once. It matches or beats continuous Riemannian Bayesian optimization in attempts-to-success. It does so at detector-level latency. And it stays stable across several orders of magnitude of step size, where fixed-step gradient refinement collapses. The ingredient that is new is the truncation. The loss we refine is a negative log-probability, so it is nonnegative with a known lower bound of zero. This is exactly the structure that Stochastic (Approximate) Proximal Point Methods exploits to obtain stability across six orders of magnitude of step sizes in Euclidean convex problems. We carry that structure onto the pose group and into a non-convex learned landscape.
Let $T$ be a gripper pose in SE(3), the group of 3D rotations and translations. Let $s(T)$ be the grasp evaluator's success logit. Let $r(T)$ and $\sigma_r(T)$ be the posterior mean and standard deviation of a Gaussian-process residual fitted to this session's grasp outcomes, using a geodesic-distance kernel as in the geometry-aware Bayesian optimization paper. Let $\beta$ be an optimism weight. The refined loss is
$$\ell(T) = -\log \sigma\big(s(T) + r(T) + \beta\,\sigma_r(T)\big) \;\ge\; 0,$$
where $\sigma$ is the logistic function. Let $g$ be the gradient of $\ell$ expressed in the 6-dimensional tangent space (Lie algebra) at $T$, and let $\alpha$ be the nominal step size. One truncated step is
$$T_{k+1} = \exp\!\Big(-\min\big(\alpha,\ \ell(T_k)/\|g\|^2\big)\, \hat g\Big)\, T_k .$$
The step can never overshoot the predicted zero of the loss, so an oversized $\alpha$ is capped by $\ell/\|g\|^2$ rather than amplified. The hard part is that the Asi–Duchi guarantees assume convexity. Whether the cap still prevents overshoot into collisions on a learned, non-convex evaluator is an empirical question, and this proposal is built to answer it.
Relation to Prior Work
Three works supply parts of the method.
- 6-DOF GraspNet samples 6-DOF grasps from a conditional variational autoencoder. It then nudges them by fixed-step gradient ascent on a differentiable PointNet++ evaluator, reaching 88% real success on isolated objects with a 0.04 s forward pass. We reuse its evaluator as $s(T)$. We replace its fixed step, and we add a session-level uncertainty model, which it lacks.
- GaBO supplies the geodesic kernels and shows that treating orientations as flat vectors breaks Bayesian optimization. It maximizes Expected Improvement from generic multistarts with Riemannian conjugate gradient. We instead start from detector proposals and never solve a free-standing continuous acquisition problem.
- aProx supplies the truncated update.
Two further works motivate the proposal by showing that the acquisition bottleneck is real.
- Automatic LQR Tuning tolerated seconds of acquisition only because each trial took two minutes.
- α-PFN amortizes Entropy Search with a pretrained transformer, running over 50x faster than GP-MCMC. It is tied to Euclidean priors of at most 16 dimensions.
An amortized SE(3) acquisition network was the obvious idea here. We dropped it for a simpler one: once a detector provides good starting points, acquisition only needs to be optimized locally, and local optimization only needs to be stable.
Two constraining works shape the design.
- An Economic Framework for 6-DoF Grasp Detection shows that pruning labels to one grasp per approach view beats dense supervision. This tells us that a sparse portfolio does not sacrifice coverage.
- GPR shows that unconstrained refinement drives fingers into collision unless gripper width is treated explicitly. We therefore refine width as a bounded seventh coordinate.
The planar line, exemplified by the Equivariant Transporter Network, sets the application regime. That work reports at least 95% success with 10 demonstrations, but only for top-down picks. We target exactly the grasps it cannot express.
Research Program
We will build the refinement loop on top of a frozen 6-DOF grasp evaluator and run it in two settings. In the offline setting, refinement uses the evaluator alone, with the residual set to zero. This isolates step-size behavior on GraspNet-1Billion scenes. In the online setting, clutter removal runs in physics simulation with ACRONYM meshes, and then on a physical Franka Panda. After every attempt, the Gaussian process is updated with the outcome at the executed pose and the portfolio is re-refined. Candidate selection among the refined portfolio uses the lowest $\ell$, which is an upper-confidence rule. A Thompson-sampling variant is included for comparison.
Beyond the pipeline, we will run two analyses. The first maps success rate against the nominal step size $\alpha$ over six orders of magnitude, for three update rules: fixed step, AdaGrad-Norm (whose self-correction is analyzed in Linear Convergence of Adaptive Stochastic Gradient Descent), and truncated steps. The second counts how often each rule pushes a candidate into collision with neighboring objects. We will also stratify all scenes by whether a top-down approach is kinematically feasible. This separates gains on grasps that are genuinely beyond planar reach.
Baselines
The naive reference is the frozen detector used as-is. It executes the top-scoring grasp and, on failure, the next-highest-scoring unexecuted grasp, with no refinement and no learning within the session. The strongest instance of this is EconomicGrasp, which reports 92.3% success in real cluttered scenes on a Franka, against 87.8% for GSNet, and a 3 AP gain over GSNet on GraspNet-1Billion. It is the strongest competitor because it already reflects the best offline learning, and any online method must beat it on the same scenes. We will run it on identical object layouts. Beating it means a statistically significant gain in first-attempt success or attempts-to-clearance on the non-top-down stratum, at a per-decision latency no more than twice the detector's.
The strongest refinement baseline is 6-DOF GraspNet's fixed-step gradient refinement. It reports 88% on isolated real objects, and we found no reported result on Lacuna for it in clutter. We give it the same number of gradient evaluations, and we tune its step size with the same grid used to show our method's insensitivity. It thus gets its best $\alpha$, while ours is judged at every $\alpha$. We also include AdaGrad-Norm refinement under identical budgets.
The strongest acquisition baseline is GaBO-style Expected Improvement over SE(3), using the same geodesic Gaussian process and multistart Riemannian conjugate gradient. For this baseline, GaBO reports roughly 1.5 s per iteration against 0.5 s for Euclidean optimization on its much smaller problems. It receives the same physical attempt budget.
Two ablations complete the set. The first sets $\beta=0$ and $r=0$, keeping refinement but removing learning. The second keeps the Gaussian process but drops refinement, selecting among raw detector proposals. Together they isolate whether the gain comes from truncation, from session learning, or from their coupling.
Why not just…?
Why not just stay planar with an equivariant policy? Equivariant planar models are extraordinarily sample-efficient, but only within top-down picking. PEGG-Net and MetaGrasp document failures when overhead approaches are occluded or unreachable. Our stratified evaluation reports results separately on scenes where a top-down grasp exists. On those scenes we expect planar methods to be competitive and will say so. The claim is confined to the stratum they cannot reach.
Why not just Thompson-sample over the detector's top-K and skip refinement? This is the real threat. Selection over a finite set is nearly free, and it may capture most of the benefit of session learning. The no-refinement ablation is exactly this. If it matches the full method on attempts-to-success, the acquisition part of our claim fails. The contribution then reduces to step-size-robust refinement for offline detection, which we would report as the result.
Why not just fine-tune the detector on the robot's failures? A handful of binary outcomes per session is far too few to move a large network safely. The EconomicGrasp results show that even offline training is sensitive to conflicting labels. A small Gaussian-process residual with tens of observations is the natural low-sample alternative. We will nonetheless include a last-layer fine-tuning arm with a matched update budget.
Why not just use AdaGrad-Norm, which already self-corrects oversized steps? Its guarantees rely on interpolation-like conditions, and it corrects only after large gradients have already inflated its denominator. In grasping, that means after a candidate has already jumped into a collision. The step-size sweep and the collision count will show directly whether its transient overshoot matters in practice.
Decisive Checks
Three outcomes would settle the proposal.
The first is the width of the stable step-size window. We measure the range of $\alpha$ over which offline refined-grasp AP on GraspNet-1Billion stays within one point of its best value. We expect roughly six orders of magnitude for truncated steps, by analogy with aProx, against one or two for fixed steps.
The second is the online comparison. We measure attempts-to-first-success and clearance rate on non-top-down simulated scenes against EconomicGrasp's execution order and GaBO-style Expected Improvement, reporting per-decision wall-clock alongside.
The third is the physical Franka replication on a fixed set of cluttered layouts, compared with the 92.3% reference.
If truncated refinement is no more robust than tuned fixed steps, or if the no-refinement Thompson arm matches the full method, the central claim is falsified. Either outcome is informative, and both are cheap to observe.
References
- Bayesian Optimization Meets Riemannian Manifolds in Robot Learning
- Automatic LQR tuning based on Gaussian process global optimization
- Stochastic (Approximate) Proximal Point Methods: Convergence, Optimality, and Adaptivity
- 6-DOF GraspNet: Variational Grasp Generation for Object Manipulation.
- $\alpha$-PFN: Fast Entropy Search via In-Context Learning
- An Economic Framework for 6-DoF Grasp Detection
- GPR: Grasp Pose Refinement Network for Cluttered Scenes
- Equivariant Transporter Network
- Linear Convergence of Adaptive Stochastic Gradient Descent
- PEGG-Net: Pixel-Wise Efficient Grasp Generation in Complex Scenes
- MetaGrasp: Data Efficient Grasping by Affordance Interpreter Network