Support-Graph Reward Shaping for Sim-to-Real Retrieval from Stacks

Picture a robot asked to fetch a mug buried under a plate and two bowls. To get the mug it has to move away from it first: lift off the bowls, then the plate, then reach the mug. A learning agent that is rewarded only when it finally holds the mug almost never stumbles onto this sequence. The usual fixes for this kind of sparse reward reward "getting closer" or "making visible progress," and those signals mislead the agent here, because the correct early actions move away from the target. Our idea is to reward the robot for shrinking one quantity: how much of the pile still physically rests on, or blocks, the target. Robotics already has learned models that predict this structure as a support graph, meaning a directed graph whose edge A→B says object A rests on or blocks object B. We turn the number of objects still above the target in that graph into a shaping signal of the form that provably cannot change which policy is optimal. That signal trains a reinforcement learning (RL) policy in simulation without a hand-built curriculum. The same quantity, computed from a graph predicted on real camera data, then becomes a dense reward for a short phase of real-robot fine-tuning that closes the remaining sim-to-real gap.

This sits where three lines of work meet without overlapping. Support-graph methods order objects well but hand the order to scripted, open-loop execution. Sparse-reward methods for robots measure progress in ways that break when the task needs detours. Sim-to-real systems for clutter get trained policies onto hardware but treat clutter as undifferentiated density. One object, the target's ancestor set in the support graph, can serve all three roles: a reward in simulation, a feature the deployed policy can predict, and a progress measure on the real robot.

Core Claim

Write $G(s)$ for the support graph of state $s$, and let $\mathrm{Anc}_{G}(t)$ be the set of objects with a directed path to the target $t$, that is, everything that rests on or blocks it, directly or transitively. We define the potential

$$\Phi(s) = -\sum_{j \in \mathrm{Anc}_{G(s)}(t)} c_j ,$$

where $c_j$ is the cost of clearing ancestor $j$. We set $c_j = 1$, except that a group of objects in "stable support" (a child that can be carried together with its parent, as in the stable/weak distinction of Prioritized Planning for Target-Oriented Manipulation via Hierarchical Stacking Relationship Prediction) counts once. The agent is trained on

$$r'(s,a,s') = r(s,a,s') + \gamma\,\Phi(s') - \Phi(s),$$

where $r$ is the sparse task reward and $\gamma$ is the discount factor. $r$ gives +1 for lifting the target and a penalty for any object leaving the workspace without being grasped or exceeding a velocity threshold, which is how we detect collapse. Because the shaping term is a difference of potentials, it leaves the optimal policy unchanged. This is the guarantee Hierarchical Potential-based Reward Shaping from Task Specifications relies on when it derives potentials from temporal-logic specifications for driving and locomotion. We borrow that guarantee and replace the specification with a scene graph.

We make three claims. First, this potential replaces the staged curriculum that clutter RL currently needs. Second, unlike time- or appearance-based progress critics, it rises during detours away from the target. Third, when it is computed from a graph predicted on real sensor data, it is a better real-world fine-tuning reward than a generalist video progress critic on tasks that need detours.

Research Program

Where each strand comes from. The support-graph strand motivates the potential. HSRN predicts stable/weak support trees with a gated graph network and plans over them with a POMDP (a planner that reasons over uncertain beliefs about the scene), reporting 95.63% planning rationality on REGRAD-v2 against 47.36% for the earlier VMRN relationship model. Learning Spatial-Aware Manipulation Ordering (OrderMind) distills a vision-language model's ordering judgments into a fast attention network. Both output an order for scripted execution, and OrderMind names the static-scene assumption as a limitation. We keep their graph and discard the open-loop executor.

The sim-to-real strand supplies the training recipe. ClutterDexGrasp trains a teacher policy that sees privileged simulator state, then distills it into a point-cloud student that transfers to hardware without real training data. It reports 83.9% real grasp success, and 0% success when trained in dense clutter without its three-stage curriculum. That 0% is the evidence that sparse rewards are the bottleneck. Its teacher knows only distances to objects, not which objects support which, so we change exactly that.

The progress-critic strand constrains us. VLAC improves real-robot RL from 30% to 90% in 200 episodes using pairwise frame-progress estimates, but it names the assumption that progress increases monotonically over time as a failure mode. That is precisely the case of clearing a blocker. SARM handles stages explicitly but needs human stage labels. Our potential is stage-aware by construction, with no annotation.

Stage 1: simulation teacher. We build retrieval scenes in simulation using REGRAD-v2's tableware categories (ShapeNet and YCB objects, 8 to 12 per scene). We add denser vertical stacks of 15 to 25 objects, with randomized friction and mass. The ground-truth support graph is recomputed every control step from simulator contacts, so it updates when objects slide or fall. This directly targets the static-scene limitation of OrderMind and HSRN.

A PPO teacher drives a parallel-jaw arm. It sees privileged object poses plus the ground-truth graph and is trained on $r'$ with no curriculum. We use a parallel jaw rather than a dexterous hand so that results isolate the reward question from hand dexterity.

Stage 2: point-cloud student. The student sees single-view point clouds. It is trained by DAgger, an imitation method in which the teacher relabels the states the student actually visits. The student also has an auxiliary head that predicts the support graph from its own features, supervised by simulation ground truth. Visual and physical domain randomization follow ClutterDexGrasp's practice. The auxiliary head matters for Stage 3 and lets us ask whether predicting structure helps transfer at all.

Stage 3: real-world fine-tuning with a structured reward. On a real UR5e-class arm we compute $\Phi$ from the student's predicted graph and fine-tune a residual policy for 200 real episodes. The residual policy is a small correction added on top of the student's actions. We use 200 episodes to match VLAC's reported budget. On real data the potential depends on an estimate, so it no longer carries the formal invariance guarantee. We therefore treat it as a learned critic and measure how its errors relate to failures. This stage also tests whether predicted graphs are good enough to serve as rewards on hardware, which is the perception-error risk both OrderMind and JENGA flag.

Baselines

Sparse reward only. The naive reference point is the identical PPO teacher trained on $r$ alone, with the same simulator, scenes, network, and step budget. ClutterDexGrasp's 0% without curriculum suggests it will barely learn on dense stacks. Its role is to establish how sparse the problem is in our scenes, not to be beaten by a narrow margin.

Curriculum teacher. The strongest existing RL method is ClutterDexGrasp's staged curriculum, which raises clutter density step by step and then adds safety constraints. We re-implement it for a parallel-jaw gripper on retrieval in our scenes, with the same total environment steps and the same hyperparameter sweep size as our method. Its reported numbers are 91.9% teacher success in simulation on seen layouts and 83.9% zero-shot on real hardware, but those are for dexterous grasping, not retrieval from stacks. For retrieval from vertical stacks with a learned policy, no reported result was found on Lacuna. Our method beats this baseline if it matches the curriculum's final success with fewer environment steps and no hand-designed stages, or exceeds its success on the deepest stacks, where target depth in the graph is 3 or more.

Closed-loop modular pipeline. The strongest non-RL competitor runs HSRN's graph predictor, trained on REGRAD-v2, after every action, then executes the first object in its plan with a standard grasp planner. This is an honest version of "use a planner," upgraded from HSRN's original setup by replanning every step. It gets the same perception input and the same grasp-candidate generator as our student. It is the baseline we most expect to be competitive on static, well-separated stacks. We count it as beaten only if our policy wins on success or collapse rate on stacks where execution shifts the pile: low-friction, top-heavy, and leaning configurations.

Generalist progress critic. For Stage 3 the baseline is VLAC used as the fine-tuning reward, with the same 200 episodes, the same residual policy, and the same starting student. Our method beats it if it reaches higher success on tasks that require one or more detours, and if the gap shrinks on tasks where the target is unobstructed. The second condition is the signature that monotonicity is the cause.

Ablations isolating the key ingredient. We train three variants with identical budgets. The first uses a privileged-state dense reward with no graph: negative gripper-to-target distance plus a count of objects removed. The second uses the graph potential with all objects counted, not just ancestors. The third uses the ancestor potential with stable support ignored. Together these separate three possible sources of benefit: the graph's directed structure, its focus on the target's ancestors, or merely the presence of any dense signal.

Why not just…?

Why not just run OrderMind or HSRN with a motion planner? On static stacks this may do about as well, and we say so up front. HSRN's 29.48% execution-time reduction on a real UR5e shows that graph-guided planning is strong when the scene stays put. The case against it rests on the dynamic setting. ClutterDexGrasp notes that open-loop grasps cannot adapt to unexpected movement, and OrderMind lists dynamic shifts as unhandled. The modular baseline above includes per-step replanning, so if it ties us on shifting stacks, the honest conclusion is that the graph matters and RL does not. We would report that.

Why not just hand-write a dense reward, since simulation knows every pose? Distance-to-target pays the agent for approaching a buried target and punishes it for lifting the plate on top. Counting removed objects rewards clearing irrelevant clutter. This alternative is a real threat, because a carefully tuned heuristic might suffice. The first ablation tests it directly, and the comparison is most telling on deep stacks.

Why not just use a curriculum, as ClutterDexGrasp does? Curricula work, but every new task family needs its stages redesigned. The potential carries the same information with no extra design work. The curriculum baseline will show whether the potential matches it, and the two can be combined if neither dominates.

Why not just use VLAC as the real-world reward? Its own authors identify detours as a failure mode, and our task is mostly detours. The Stage 3 comparison, split by number of required detours, settles this question.

Proposed Experiments

Simulation. We report retrieval success, collapse rate, and environment steps needed to reach a fixed success level, broken down by the target's depth in the support graph. If the potential does what we claim, the gap to sparse-only training should widen with depth. If gains appear only at depth 1, the claim fails.

Transfer. The real evaluation uses about 40 stacked tableware scenes of graded difficulty, reporting success and disturbance for zero-shot transfer from each teacher variant. We also run a student trained without the graph head, to test whether predicting structure aids transfer.

Decisive checks. Two checks decide whether the real-world claim holds. First, Stage 3 must beat VLAC specifically on detour tasks. Second, the accuracy of the predicted graph should predict when graph-reward fine-tuning succeeds. If the predicted graph is too noisy on real data, for example from errors on lower, occluded objects like those Park et al. observe, the real-world reward claim fails even if the simulation claim holds. We would report the two results separately.