Branching on Contact Outcomes: Stochastic Mode Graphs for Multi-Stage Manipulation

Gradient-based optimization through a physics simulator can tune a robot's motion in tens of iterations instead of thousands of trial-and-error episodes. It does this well only while contact stays the same: within a single press or slide, it works. When a task requires making, breaking, and remaking contact, it stalls. Real contact is also not deterministic. Whether a box pivots or slides, or whether a gripper sticks or slips, depends on friction and geometry the robot cannot observe. The outcome often splits into a few distinct possibilities rather than spreading out as smooth noise. We propose to treat those two facts as one structure. The low level is smooth and nearly deterministic inside a contact mode, meaning a fixed set of touching surfaces with fixed stick or slip status, and gradients serve it well. The high level is a small set of discrete contact outcomes, where randomness actually lives, and it should be planned over as an explicit branching probability.

The concrete system is a graph. Its nodes are contact modes, read automatically from the simulator. Its edges are short gradient-optimized motion segments, and each edge carries a learned categorical distribution over which mode it actually lands in. Search over this graph is restricted to edges whose estimated probability of eventually reaching the goal, without exceeding force or torque limits, clears a threshold. Expensive low-level optimization is therefore spent only where it can pay off. The claim is that this factorization beats two alternatives on multi-stage contact tasks with randomized physical parameters. The first is a single smoothed model optimized end to end. The second is a deterministic hybrid planner. We expect the advantage to grow as the outcomes at contact switches become more bimodal.

What the literature establishes and where it stops

Three papers motivate the work by documenting why flat gradients fail through contact. End-to-End Differentiable Physics for Learning and Control shows that a gripper hovering just above an object receives zero gradient. Do Differentiable Simulators Give Better Policy Gradients? shows that first-order gradients become biased near unmodeled boundaries and explode in stiff settings, and it proposes blending them with sampling-based estimates. PlasticineLab documents the "multi-stage wall": Adam reaches 0.93 normalized IoU (intersection over union with the target shape) on RollingPin almost immediately, yet stalls on press–release–press tasks such as Pinch. PlasticineLab explicitly calls for a hybrid in which a discrete layer switches contact modes and gradients handle continuous deformation, but it does not build one.

Two lines supply our low-level method. Bundled Gradients through Contact via Randomized Smoothing averages gradients over a local Gaussian neighborhood so that a robot can "feel" contact before touching. On the Drake pushing and pivoting tasks it succeeds where exact-gradient MPC (model predictive control, i.e. repeated short-horizon replanning) fails from non-contact starts. PWM shows that a deliberately smoothed learned world model optimizes better than exact differentiable physics. We use bundled gradients for mode-entering segments and plain first-order gradients inside a mode. Both lines smooth everything indiscriminately, though. In doing so they blur exactly the bifurcations that decide whether a multi-stage plan succeeds.

The closest existing work is Lazy INSAT. It interleaves A* search over configurations with trajectory optimization through contact, and it evaluates edges lazily, optimizing only those the search shortlists. It assumes known friction and deterministic outcomes, and it takes about 3 minutes per plan. Our graph differs in three ways. Its nodes are contact modes rather than configurations. Its edges are stochastic, with learned outcome distributions. Its expansion is pruned by a reachability value rather than by a cost heuristic.

The pruning rule comes from Feasible Reachable Policy Iteration. That paper restricts exploration to states that are both safe and able to reach the goal, and it reports convergence 3–5 times faster than SAC-Lagrangian on Safety Gym. It needs hand-built goal functions, and it operates on primitive actions in smooth navigation tasks. We apply its recursion on the mode abstraction instead, where goal and violation sets can be read directly from simulator contact and force signals.

On the stochastic side, STEVE and Self-Supervised RL that Transfers using Random Features show that mean-state rollouts drift and that ensembles help. The file also notes that Gaussian transition models are uncalibrated at contact boundaries, where outcomes branch. That observation is the reason we model outcomes as categorical over modes and do not use continuous variance.

Formal shielding, as in Safe Reinforcement Learning via Shielding, and offline hybrid reachability, as in Formal Verification of Robotic Contact Tasks via Reachability Analysis, act as constraints on what we can claim. The first requires coarse deterministic abstractions. The second verifies impacts rigorously but trains nothing. We give up formal guarantees in exchange for a learned, statistically estimated reachability value, and we say so plainly.

The hierarchical-control literature, including SuSIE and HiRT, is relevant through one failure: SuSIE's high level proposes subgoals the low level cannot reach. Our design addresses this directly. The high level only ever chooses among subgoals, here target modes, whose reachability has been measured by running the low-level optimizer. We do not claim to solve language-conditioned planning.

Core Claim

Randomness in contact-rich manipulation is concentrated at mode switches and is often multimodal there. A planner should therefore represent outcomes as a discrete distribution over next modes, and leave the continuous motion within a mode to deterministic gradient optimization. Under this factorization, a reachability value over the mode graph can be computed by a short recursion. That value both selects robust plans and prunes exploration. The result should beat flat smoothing on multi-stage success, and beat deterministic hybrid planning on success under randomized physics, at equal simulator budget.

Research Program

Modes are extracted automatically. In rigid-body tasks, a mode is the set of active contact pairs together with the stick or slide status of each. In PlasticineLab, a mode is the set of tools in contact with the material.

An edge from mode $m$ consists of segment parameters $\theta$ and a target mode. The segment is optimized by bundled gradients while contact is being established, and by first-order gradients through the simulator once inside the mode. After optimization, the segment is executed under $K$ draws of the randomized physical parameters (friction, mass, material stiffness). The observed landing modes train a small classifier $p_\phi(m' \mid s, \theta)$ that conditions on the start state $s$.

The reachability value over modes satisfies

$$V(m) = \max_{e \in E(m)} \sum_{m'} p_\phi(m' \mid e)\, V(m'), \qquad V(\text{goal}) = 1,\; V(\text{violation}) = 0,$$

where $E(m)$ is the set of optimized edges leaving $m$. A violation is any mode reached with force or torque above a limit. $V(m)$ is then the estimated probability of reaching the goal safely from $m$. This is the reachability recursion of FRPI restated on a finite stochastic abstraction.

Search expands an unoptimized candidate edge only if an optimistic estimate of its value exceeds a threshold $\tau$. For that estimate we use an upper confidence bound, computed with the classifier's ensemble disagreement as epistemic width. This pruning rule is the constrained-exploration ingredient.

At execution time, the robot follows the maximizing edge and observes the landing mode. It then continues from that branch rather than replanning from scratch, because the graph already contains the edges leaving each likely outcome.

The analysis question is when branching beats robust averaging. Our hypothesis is that it does so when $p_\phi$ is bimodal and the best continuation differs between branches. We will measure outcome entropy at each edge and correlate it with the gap between methods.

Baselines

The naive references are two methods reported in PlasticineLab: full-horizon Adam through the differentiable simulator, and model-free PPO. On RollingPin the paper reports 0.93 IoU for Adam and roughly 0.80 for PPO. For Pinch it reports only that gradients stall, and we found no reported Pinch IoU on Lacuna. Our first measurement will therefore be these baselines on Pinch under the original deterministic physics. We need it to calibrate everything else.

The strongest flat competitor is full-horizon randomized-smoothing optimization, iRS-MPC from Bundled Gradients through Contact via Randomized Smoothing. It is the method most likely to cross contact boundaries without any hierarchy. The paper reports success from non-contact starts on planar pushing and box pivoting with N=100 samples per knot point, but no success rate under randomized friction; we found no reported result on Lacuna for that setting. We will give it the same samples-per-step budget, and extend its Gaussian to cover parameter randomization.

The second strong competitor is PWM-style first-order policy learning through a smoothed learned world model, PWM. Its reported advantage is up to 27% higher reward than TD-MPC2 on MT80, a different suite from ours. We will train it on the same randomized-physics data our classifier sees.

The closest structural competitor is a deterministic mode graph in the style of Lazy INSAT. It uses the same modes and the same low-level optimizer, but optimizes each edge for the expected loss over sampled parameters and treats the most likely landing mode as certain. We give it replanning on failure.

Two ablations isolate our ingredients. One removes the reachability threshold, so that all edges are expanded. The other replaces the categorical outcome model with a Gaussian ensemble over continuous end states, in the style of STEVE. For constrained exploration we also run FRPI-SAC over primitive actions, to test whether pruning without the mode abstraction suffices.

Fairness is enforced by a shared budget: every method gets the same number of simulator steps, with each backward pass counted at its measured wall-clock cost in forward-step equivalents. Every method also gets the same number of hyperparameter trials and the same randomized parameter distribution at test time.

Beating the field means two things. First, higher success on held-out physics draws than iRS-MPC and PWM on multi-stage tasks. Second, against the deterministic mode graph, either higher success or the same success with fewer optimizer calls, and with fewer limit violations during search in either case.

Why not just…?

Why not just smooth the whole horizon with randomized smoothing? Smoothing makes contact visible before it happens, but it also averages over the branch point. When a pivot either succeeds or slides depending on friction, the averaged gradient points toward a hedge that does neither well. Nothing in the file shows bundled gradients clearing a press–release–press sequence. This is the most serious threat, and the comparison on Pinch and on bimodal pivoting will settle it. If iRS-MPC matches us, the hierarchy is not earning its cost.

Why not just train a smoothed world model and backpropagate, as PWM does? PWM shows that smoothing helps optimization, but its policies explore without constraint, and its continuous latent dynamics cannot represent a two-outcome contact event with calibrated probabilities. The Gaussian-ensemble ablation and the PWM baseline together test whether discrete outcome modeling matters. We predict the gap closes when parameter variance is set to zero, and if it does not, our explanation is wrong.

Why not just use a deterministic hybrid planner with domain randomization and replanning? This is a real threat. Fast replanning after observing the landing mode could recover most of the benefit of branching. Our hypothesis is that the deterministic planner wastes optimization on edges whose success probability is low, and fails when no continuation exists from the unlucky branch. We will report success as a function of outcome entropy and the count of optimizer calls. If the deterministic graph with replanning ties on both measures, we will conclude that only the reachability pruning, not the stochastic model, is the contribution.

Why not just use model-free constrained RL? PlasticineLab reports PPO at roughly 0.80 IoU on RollingPin against 0.93 for Adam, and FRPI reports SAC-Lagrangian converging 3–5 times more slowly than reachability pruning. We include FRPI-SAC rather than argue the point further.

Proposed Experiments

The benchmarks are PlasticineLab's multi-stage tasks (Pinch, TripleMove) and the Drake suite (planar pushing, box pivoting, two-fingered hand). In each, we randomize friction and material parameters per episode, sweeping the variance from zero upward. For pivoting, we center friction near the stick–slide threshold so that outcomes are bimodal. These randomized variants are our own modification; the original benchmarks are deterministic.

The metrics are four. Success rate or IoU, measured on held-out parameter draws. Simulator-step-equivalents to first success. Force and torque limit violations during search and execution. Calibration of $p_\phi$, measured as expected calibration error on landing modes.

The decisive checks are three. At zero variance, the stochastic and deterministic mode graphs should tie, and both should beat full-horizon Adam on Pinch. As variance grows, the success gap over the deterministic graph should grow, and it should track measured outcome entropy. Removing the reachability threshold should leave success unchanged while raising optimizer calls and violations. If any of these predictions fails, the corresponding ingredient is not doing the work we claim for it.