Geometry Heat Kernels as Texture-Free Targets for Sim-to-Real Vision

A robot policy trained in simulation usually fails on real hardware for a mundane reason. Its camera encoder learns to rely on how simulated surfaces look, such as uniform shading, repeated textures and clean lighting, rather than on the shapes that determine where to grasp or where to fly. The standard fix, visual domain randomization, repaints the simulated world with random textures and lighting so that appearance stops predicting the task. This works partly, but it only removes texture cues by averaging over them. Nothing in the training signal tells the encoder what it should represent instead.

We propose to supply that missing positive signal from the simulator's geometry. For every simulated frame we know the 3D position of each pixel. From these positions we build a graph whose edges depend only on 3D proximity and surface continuity and never on color. We then compute its heat kernel, a matrix that says how strongly two points are connected through the surface at several scales. We train the RGB encoder so that inner products between pixel features reproduce this heat kernel, and we use the same target for every texture randomization of a scene. Because the graph is built from geometry alone, the target cannot be satisfied by texture. At deployment the robot needs only an RGB camera. The claim is that a geometry-derived spectral target of this kind gives zero-shot RGB policies a texture-invariant representation more reliably than randomization alone, pixel-level translation, or simpler geometric auxiliary targets such as depth.

Core Claim and Why It Is New

Spectral graph embedding places the nodes of a graph in space using the eigenvectors of its Laplacian, the matrix that measures how much a signal varies across edges. Nodes that are densely connected then land close together, a construction formalized as Laplacian Eigenmaps. Applied to a graph over a scene's 3D surface points, the smoothest eigenvectors separate objects that do not physically touch, and the next ones separate the parts within an object. This is the relational, multi-scale structure a grasping or navigation policy needs, and it is independent of paint by construction.

Using eigenvectors directly as regression targets is fragile. They have arbitrary signs, rotate freely within repeated eigenvalues, and swap order under small perturbations. The research file lists this "eigenvector switching" as an open problem for spectral methods. We therefore match the heat kernel $K_t = \exp(-t L_{\text{sym}})$, which does not depend on the choice of eigenvector basis. Here $L_{\text{sym}}$ is the degree-normalized Laplacian of the geometry graph and $t$ is a diffusion time that sets the scale.

The closest prior work divides into three groups. The first group turns images into graphs for shape bias. Shape-Biased Domain Generalization via Shock Graph Embeddings converts 2D contours into medial-axis graphs and classifies them with a message-passing network. It needs clean edge maps at test time, which cluttered real scenes break, and it never touches control. Our graph exists only at training time, inside the simulator, where geometry is exact. At test time we run an ordinary image encoder.

The second group learns structure implicitly from appearance variation. S2R-DepthNet trains a structure encoder against painting styles for zero-shot depth estimation. Sim-To-Real via Sim-To-Sim (RCAN) translates randomized or real images into a canonical rendering before grasping. Both still express structure in pixel space, and RCAN inherits generator artifacts. We supervise relational geometry directly.

The third group randomizes appearance without any structural target. This includes Asymmetric Actor Critic, (CAD)²RL and Conditional Patch-Based Domain Randomization. CPDR in particular enriches the texture distribution with real patches rather than removing texture dependence. These methods motivate the proposal: they show randomization helps but must discover invariance by brute force. We reuse their randomization as the source of varied views, and add a target that is fixed across those views.

Two spectral papers supply parts of the method but not the application. Contrastive Laplacian Eigenmaps shows that Laplacian-derived losses keep useful gradients where sigmoid contrastive losses stall, but only on citation graphs. Transformers as Unrolled Inference in Probabilistic Laplacian Eigenmaps reads attention as graph diffusion, but never studies domain shift. Neither builds the graph from simulator geometry or uses it as a target across a reality gap.

Research Program

For each rendered frame we sample $N \approx 1024$ pixels and back-project them to 3D points $p_i$ using the simulator's depth. We connect each point to its $k$ nearest 3D neighbors with weight $W_{ij} = \exp(-\|p_i-p_j\|^2/\sigma^2)$. We cut an edge when the depth jump or the surface-normal disagreement between the two points exceeds a threshold, so that touching but distinct surfaces stay weakly linked. From $W$ we form $L_{\text{sym}}$. We compute $K_t$ by one eigendecomposition per frame for three diffusion times $t$. At $N=1024$ this costs about a billion floating-point operations on a GPU, and subsampling pixels is how we sidestep the cubic cost of exact decomposition.

The encoder produces features $f_i$ at those pixels, with one projection head $g_t$ per scale. It minimizes

$$\mathcal{L} = \sum_{t} \big\| G_t - K_t \big\|_F^2, \qquad (G_t)_{ij} = \langle g_t(f_i), g_t(f_j) \rangle,$$

summed over every texture and lighting randomization of the same geometry. $G_t$ is the feature Gram matrix at scale $t$, and $\|\cdot\|_F$ is the Frobenius norm.

The encoder is first trained with this loss on randomized renders. It then serves as the actor's visual backbone in an asymmetric actor-critic setup, following Asymmetric Actor Critic, where the critic sees the full simulator state. We compare two variants: one freezes the encoder, and one fine-tunes it jointly with a small weight on $\mathcal{L}$.

The analysis has three parts. First, we measure how much of the encoder's feature variance is explained by texture identity versus geometry, using held-out scenes where the textures are swapped between objects. This is a cue-conflict test in the spirit of Geirhos et al.. Second, we measure how much the policy's actions change under texture swaps. Third, we track how both quantities evolve over training.

Baselines

The naive reference point is a policy trained on canonical, unrandomized simulation images and deployed directly. It measures the raw reality gap on our hardware.

The standard competitor is visual domain randomization with the same encoder and training budget but no geometric loss. This is the exact configuration our method adds to. In the RCAN grasping evaluation, standard randomization reached 33–37% zero-shot real grasp success (RCAN).

The strongest existing method on this benchmark is RCAN, which reports 70% zero-shot real grasp success on its Kuka setup. Because that number comes from different hardware, we reimplement RCAN on our arm with the same randomized scene generator and object set. We give its translation network the same GPU-hours we spend on encoder pretraining. Beating RCAN means exceeding the reimplemented RCAN's zero-shot success on the same real objects over at least 100 grasps per method, with non-overlapping binomial confidence intervals. We also report whether our absolute rate approaches the published 70%.

For perception, the strongest competitor is CPDR on YCB-M detection and segmentation. The paper reports gains of 4.94 and 6.71 AP over the best prior randomization for its conditional Patches B variant (CPDR). The file gives no absolute AP, so we rerun CPDR ourselves. We also test whether our loss adds to CPDR when the two are combined.

Three ablations isolate the key ingredient, and all use identical data, encoder, and tuning budget, with the same number of hyperparameter trials per method. The first builds the graph with color similarity mixed into the edge weights; if texture-free edges matter, this should hurt. The second replaces the heat kernel with direct regression onto sign-aligned eigenvectors; if basis-free targets matter, this should be less stable. The third replaces the spectral target with per-pixel depth-and-normal regression or instance-mask prediction, both of which the simulator also provides for free.

Why not just…?

Why not just predict depth and normals as an auxiliary task? This is the most serious threat, because the simulator gives depth for free and it too is texture-independent. Our reason to expect a difference is that depth is local and metric. Two touching objects at the same depth look alike in a depth map, but they are weakly linked in the cut geometry graph. The heat kernel encodes exactly this contact and part structure, which matters for grasp selection. This is a hypothesis, not established evidence. The third ablation settles it directly. If depth regression matches our real grasp success and texture-swap stability, we will report that the spectral target adds nothing beyond geometry supervision.

Why not just give the policy a depth camera? For opaque objects, a depth-input policy may be strong, and we include one as an additional reference. It carries its own sim-to-real gap from sensor noise and missing returns on shiny or transparent surfaces. It also does not help RGB-only platforms such as the monocular drone in (CAD)²RL. If depth input clearly wins on our object set, we narrow our claim to RGB-only settings rather than argue around the result.

Why not just randomize harder or translate to canonical images like RCAN? Heavier randomization turns a single task into a large multi-task problem that requires more capacity and longer training, as the research file notes. It also still offers no positive target for what to represent. RCAN's pixel generator introduces boundary artifacts that the downstream policy must tolerate. Our head-to-head comparison at matched compute tests whether a feature-space geometric target beats pixel-space canonicalization. We also plot real grasp success against the number of randomized frames, to test whether our method reaches a given success rate with fewer frames.

Decisive Experiments

The primary test is zero-shot real grasping on a PyBullet-to-real setup modeled on the RCAN procedural and ShapeNet object protocol. Every method is trained without any real images. We include a held-out set of real objects chosen to be adversarial to texture: similar shapes with different prints, and different shapes with similar prints. The prediction is that our encoder's success rate drops less than the randomization and depth-auxiliary baselines when moving from ordinary to adversarial objects.

The second test transfers the frozen encoder to YCB-M detection and segmentation against CPDR. It asks whether the representation generalizes beyond control.

The third test is a probing check on real indoor scenes. We probe the encoder, trained on SUNCG-style synthetic scenes, for NYU Depth v2 depth. The reference point is S2R-DepthNet's zero-shot absolute relative error of 0.196 on the SUNCG-to-NYU transfer (S2R-DepthNet). A linear probe on our features approaching that value would show that relational geometric supervision recovers metric structure in real scenes without being trained to predict depth.

The proposal fails if either of two things happens: color-mixed graphs perform as well as texture-free ones, or depth regression matches heat-kernel matching on real grasping. Either result would locate the benefit in geometry supervision generally rather than in the spectral target.