Moving Surface-Contact Demonstrations Between Object Shapes

Many useful robot skills are motions along an object's surface rather than grasps: wiping the inside of a bowl, tracing a seam, polishing a handle, scraping a pan. Collecting teleoperated demonstrations for every new object shape does not scale. The data generators that currently multiply demonstrations, such as MimicGen, move a recorded trajectory rigidly with the object. That works when only the object's position changes. It breaks when the shape changes: on a deeper bowl or a more curved handle, a rigidly moved wiping path either floats in the air or cuts into the surface.

We propose to record a contact demonstration in the object's own surface coordinates. Instead of "the tool tip went to point (x, y, z)", the demonstration becomes "the tool slid along this curve on the surface, heading in this direction relative to the surface, tilted this much, pressing this hard." We then carry that description to a new object using a functional map, a correspondence between two surfaces computed from their vibration modes rather than from matching individual points. The specific choice is a complex functional map. Unlike the standard version, it also maps directions on the surface (tangent vectors) and cannot confuse an object with its mirror image. The result is a generator that turns a handful of contact demonstrations into thousands of shape-adapted ones, which we test by training imitation policies on them.

Core Claim

The claim is that motion direction along the surface is the part of a contact skill that current transfer methods lose, and that an orientation-aware surface correspondence can carry it. Surface-contact trajectories are best transferred as a surface curve plus a heading field defined on the object's tangent planes. Pushing the heading field forward with a complex functional map should preserve contact, coverage, and tool orientation across shape changes that are not mere bending ("non-isometric" changes), where rigid replay fails. It should also fix the left/right and front/back flips that scalar surface correspondences produce.

The closest work does one half or the other. Affordance Transfer Across Object Instances via Semantically Anchored Functional Map (SemFM) maps a static contact region from a human demonstration onto a new mesh. It uses Laplace–Beltrami functional maps, which are correspondences built from the surface's scalar vibration modes, anchored by vision-language features to prevent symmetry flips. It names temporal trajectories as out of scope. Complex Functional Maps: A Conformal Link Between Tangent Bundles supplies our method. It extends functional maps to tangent vector fields through the connection Laplacian, the analogue of the Laplacian for direction fields. It reports roughly five-times lower vector-field transfer error on noisy FAUST human scans and removes flips by construction, but it was only evaluated on graphics benchmarks. On the robotics side, From Human Hands to Robot Arms: Manipulation Skills Transfer via Trajectory Alignment (Traj2Action) uses free-space Cartesian endpoint paths as its shared representation and explicitly leaves rotation alignment unsolved. MimicGen moves object-relative trajectory segments by rigid transforms. Both motivate the gap: neither ties the motion to the object's surface.

Research Program

Representation. A contact phase of a demonstration on source mesh $M$ is converted into a tuple. The first element is a surface curve $x(t) \in M$, obtained by projecting the tool tip onto the surface. The second is a unit tangent heading $h(t) \in T_{x(t)}M$, the direction of travel. The remaining elements are tool tilt angles $\theta(t)$ in the local frame formed by the surface normal and the heading, a normal offset $d(t)$, and a force profile $f(t)$. The sparse headings along the curve are spread into a smooth tangent vector field $H$ on $M$ by minimizing the connection-Laplacian (Dirichlet) energy subject to matching $h(t)$ on the curve. Approach and retreat motions in free space keep MimicGen-style object-relative handling, so the novelty is confined to the contact phase.

Transfer. We first compute a scalar functional map $C$ between $M$ and target $N$, refined to a pointwise map with ZoomOut as in SemFM, and use it to place the curve $x(t)$ on $N$. We then compute the complex functional map $Q$ linked to $C$. Writing $\hat H$ for the coefficients of $H$ in the first $k$ eigenvectors of the connection Laplacian, the target heading field is

$$\hat H_N = Q\,\hat H_M ,$$

which is evaluated along the mapped curve to give $h_N(t)$. Because $Q$ is complex-linear and respects the surface normal, a mirror-reversed heading is not an admissible output. This is what lets us drop SemFM's semantic anchors whenever the target is not a genuine reflection of the source. Tilt, offset, and force are reattached in the target's local frame. Timing is rescaled by the ratio of geodesic arc lengths, so tool speed along the surface is preserved. As in MimicGen, candidates are rolled out in simulation and kept only if they succeed.

Known constraints. Complex functional maps assume near-conformal (angle-preserving) maps. The research file lists extreme non-isometric distortion as a known failure of spectral matching. We will therefore measure performance as a function of shape distortion rather than assume it holds everywhere. Real meshes are imperfect, as noted by Boscaini et al.. To test this, we run the entire pipeline on meshes produced by Subsecond 3D Mesh Generation for Robot Manipulation, which reconstructs metric-scale meshes from one RGB-D image in 824 ms with 0.45 mm Chamfer distance on YCB. We compare against ground-truth CAD meshes.

Benchmark Construction

The research file contains no benchmark for transferring surface-following skills across shapes. We therefore build one in ManiSkill2, using its YCB and EGAD assets, supplemented with RoboCasa kitchen assets (RoboCasa). The benchmark has four task families: wiping a marked region on the inside of a container, tracing a rim or seam, brushing along a handle in a fixed direction, and scraping a curved surface. For each family we teleoperate 10 source demonstrations on 2 source objects. Targets are 200 held-out shapes, binned by a distortion score: the mean log area-distortion of the reference map. Some targets include near-mirror pairs, such as left- and right-handed tools, where the correct heading differs.

We report two levels of metrics. Trajectory-level metrics are the fraction of contact-phase time in contact, coverage IoU of the intended region, heading error against an annotated reference direction, the flip rate, and penetration depth. Policy-level metrics come from a Diffusion Policy trained on 3,000 generated demonstrations per family, matching RoboCasa's synthetic budget. We measure its success on held-out shapes, both in simulation and in a small real-robot set of 20 objects.

Baselines

The naive reference point is MimicGen-style rigid replay (MimicGen). The contact segment is expressed in the object frame and moved by the target's pose, with success filtering. It is the incumbent data generator for this kind of scaling. The closest published scale-up evidence is RoboCasa's atomic-task success rising from 28.8% with 50 human demonstrations to 47.6% with 3,000 MimicGen demonstrations (RoboCasa). No reported result was found on Lacuna for surface-following transfer across shapes, and none for MimicGen's raw success percentages. We will therefore measure this baseline ourselves on identical tasks and targets.

The strongest simple competitor is rigid alignment plus closest-point projection. The target mesh is registered to the source by ICP. Each waypoint is snapped to the nearest target surface point. Orientation is recomputed from the target normal while keeping the source heading in world coordinates. This baseline is cheap and plausibly strong when shape change is mild, so it gets the same meshes, the same success-filtering budget, and the same number of rollouts.

A second strong competitor is extrinsic non-rigid warping. A smooth space deformation is fitted between the two point sets, the Cartesian path is pushed through it, and the result is projected onto the surface. This is the natural extension of Traj2Action's Cartesian representation to shape change. It gets the same tuning budget as our method, a grid over deformation smoothness.

The key ablation replaces the complex map $Q$ with the scalar map alone, as in SemFM. The curve is mapped pointwise and the heading is recovered by finite differences of the mapped curve. This isolates whether the tangent-bundle transfer, rather than the surface curve, is responsible for any gain. A second ablation adds SemFM's vision-language semantic anchors to the scalar map. This tests whether expensive anchors buy what $Q$ provides for free.

Beating the baselines requires two things on the high-distortion and mirror-pair bins. First, a statistically significant improvement over closest-point projection in contact fraction and coverage IoU. Second, a lower flip rate than the scalar-map ablation. Downstream, policy success on held-out shapes must exceed what policies reach when trained on equal numbers of rigid-replay and projection demonstrations. On the low-distortion bin we expect parity, and we will report it as parity.

Why not just…?

Why not just snap the rigidly moved trajectory onto the new surface? This is the most serious threat. For bowls that differ only in size, or slightly curved handles, closest-point projection may be enough. It fails in two predictable ways. Nearest-point snapping jumps across thin features and concave regions, so a wiping path on the inside of a deep bowl lands on the rim. And a heading fixed in world coordinates stops meaning "along the handle" once the handle bends. The distortion-binned benchmark is built to show where, if anywhere, the gap opens. If projection matches us across all bins, the honest conclusion is that intrinsic transfer is unnecessary for these tasks, and we will report that.

Why not just use 2D semantic correspondence per waypoint, as in Robo-ABC? Robo-ABC maps contact points through diffusion-model image features and reaches 60.7% zero-shot success on cross-category affordances. However, it predicts isolated contact points. Mapping a dense sequence of waypoints independently through image features gives no guarantee that the result is a connected curve on the surface, or that the direction of travel is consistent. SemFM reports a related problem in the vision-language-model setting: GPT-4o produced scattered, disconnected regions and needed 45–53 s per object, against 9–11 s for functional maps. We will include a per-waypoint image-feature mapping run and measure curve continuity and heading error directly.

Why not just train a policy on point clouds with more rigid-replay data, or learn a trajectory editor with reinforcement learning as DemoGrasp does? DemoGrasp shows that a learned single-step edit of one demonstration generalizes grasps widely, reaching 95% Shadow Hand success on DexGraspNet. But a grasp is one pose, while a surface-following skill is a whole path whose quality depends on staying in contact over time. Learning that edit would need a reward for coverage and contact on every target. We treat this as an upper-bound comparison on one task family. We train an RL editor that outputs per-segment corrections to rigid replay, with an equal compute budget, and compare generated-data quality and downstream policy success. If the RL editor wins, our method's remaining value is that it needs no per-target training.

Decisive Checks

The first check is the mirror-pair test. On left- and right-handed targets, the scalar map should flip the heading in a measurable fraction of cases, and the complex map should not. If the complex map still flips, the core mechanism is not doing its job.

The second check is the distortion curve. Contact fraction plotted against the distortion score should decay more slowly for our method than for projection or warping. It should also show where the conformal assumption finally breaks, which we report as the method's operating range.

The third check is reconstruction robustness. Rerunning transfer on meshes from the subsecond reconstruction pipeline should cost little relative to CAD meshes. If reconstruction holes destroy the eigenvector basis, real-world use requires a repair step, which becomes the next problem.

The final check is the policy test on real objects. A policy trained only on our generated demonstrations should outperform one trained on rigid-replay demonstrations of equal count, on surfaces the robot has never touched.