Calibration-Conditioned Skip Connections for Dense Touch from Replaceable Magnetic Skins

Thin magnetic tactile skins are the most practical way to put touch on a robot finger. A sheet of rubber a few millimetres thick, filled with magnetic particles, sits over a small board of magnetometers, and the skin can be peeled off and replaced in seconds. These skins report only a dozen or so numbers per reading, so on their own they say where something pressed but not what shape pressed. Recent work showed that a network trained on paired data can turn those few numbers into a dense depth image of the contact. The pairs come from a magnetic skin and a camera-based tactile sensor that share the same contact geometry. The trouble starts in deployment: the skin wears, warms up, or gets swapped for a new one. The mapping from magnetic readings to shape then shifts, and dense reconstruction has never been tested under those conditions.

This note proposes a specific architecture and a specific test. The architecture is a U-Net, an encoder-decoder network whose "skip connections" copy intermediate feature maps from the encoder straight to the matching decoder stage so fine spatial detail is not lost. It predicts the dense contact depth map from the magnetic readings. Each skip connection is rescaled by a small code computed from a handful of standard calibration presses on the current skin. The hypothesis has two parts. First, skip connections are what let a network recover sharp contact edges from sparse sensors. Second, they are also the path through which skin-specific drift reaches the output. Conditioning them on calibration presses should keep the sharpness and remove the drift. The note also sets out a decomposition experiment that could show the whole mechanism is unnecessary, because simply re-zeroing and rescaling the raw sensor channels may already absorb most of the drift.

Core Claim

For sparse-to-dense tactile reconstruction on replaceable magnetic skins, the drift introduced by a skin swap, wear or temperature is not a per-channel offset and gain on the raw readings. It changes how magnetic readings relate to contact geometry at several spatial scales. A U-Net whose skip features are modulated by an embedding of about nine calibration presses will match the best in-distribution fidelity of existing methods. It will also lose much less fidelity on unseen skins, worn skins and warm skins than three alternatives: an unconditioned U-Net, the same U-Net with input-level recalibration, and the latent-variable model of SuperMag.

Where This Sits in Prior Work

The closest work is SuperMag. It co-designs a magnetic sensor and a camera-based sensor with identical contact modules, records synchronized pairs, and trains a conditional variational autoencoder (a generative model that squeezes the input through a global latent code) to predict depth maps. It reports 24.24 ± 2.88 dB PSNR and 0.78 ± 0.06 SSIM at 2.5 ms per reading. SuperMag provides our data-collection method and our strongest baseline. It leaves two things open. It is weak on sharp edges and fine texture, and it assumes the skin used at test time matches the training skin. We keep its paired-rig idea and change both the network and the evaluation regime.

FEATS motivates the choice of network. It shows that a U-Net can map GelSight Mini images to dense normal and shear force maps in about 4.17 ms. Its evaluation across four physical units is the only multi-unit dense-prediction evidence in the file. However, its input is already a dense camera image, and it never tests worn or heated elastomers. We move the U-Net to a sensor whose input is sparse, and to conditions where drift is severe.

The magnetic-skin papers define the drift problem. ReSkin reports that an unadapted model drops to about 25% localization accuracy on an unseen skin. Its self-supervised triplet-loss adaptation, which uses several hundred unlabeled pokes, recovers this to 87%. AnySkin tightens manufacturing through pulse magnetization and finer particles, yet still sees a 13% policy drop on a new unit. Both papers predict coordinates or labels with small multilayer perceptrons, not dense geometry. They constrain our proposal because they show that better hardware alone does not remove the shift.

The conditioning idea comes from optical tactile work. SITR feeds 18 calibration presses as prompt tokens to a Vision Transformer trained on a million synthetic images. CTSRL rescales features according to sensor identity. Both work only on camera images. The file also notes that their invariance objectives tend to discard fine texture as if it were sensor noise. We reuse the calibration-press idea but apply it inside the multi-scale skip pathway, which is where fine detail travels.

Two further papers shape the design. The barometric sensor of De Clercq et al. shows that an analytic Gaussian fit localizes a single contact to 0.51 mm median error without learning, but it cannot handle multiple or irregular contacts. The air-chamber sensor of Improving Robotic Tactile Localization Super-resolution reaches 0.13 mm centroid error from four barometers, but it recovers trajectories, not shapes. Both indicate that when the goal is only a centroid, dense reconstruction is unnecessary. We therefore evaluate on contacts with shape: an Allen key, a letter, and multi-contact probes.

Research Program

The data come from a rebuilt version of SuperMag's open-source paired rig, extended along three drift axes that SuperMag never varied. The first axis is skin identity: at least eight separately cast skins from the same recipe. The second is wear: the same skins recorded again after scheduled blocks of repeated shear loading. ReSkin reports stable accuracy after 50,000 interactions, so wear blocks run beyond that point. The third is temperature: skins recorded at several controlled ambient temperatures. Every condition uses the same indenter set and the same camera-based ground truth. The rig also records nine fixed calibration presses, a ball pressed at a 3×3 grid of positions. This is roughly half of SITR's prompt, chosen so that a robot could perform it against a fixture in under a minute.

Magnetic readings are not an image, so the network first lifts them into one. A small learned layer maps the raw magnetometer vector to a coarse feature grid. A convolutional encoder builds features at several scales from that grid. A decoder upsamples to the depth-map resolution and receives a skip connection from each encoder level. The calibration presses pass through a shared encoder and are pooled into a skin code $c$. At every skip level $\ell$, the copied feature map $h_\ell$ is replaced by

$$\tilde h_\ell = \gamma_\ell(c)\odot h_\ell + \beta_\ell(c),$$

where $\gamma_\ell$ and $\beta_\ell$ are small per-channel functions of the skin code, and $\odot$ is elementwise multiplication. During training, the calibration presses and the test contacts always come from the same skin and condition. The network therefore learns to read drift from the presses rather than memorize a skin. The whole model has to fit within SuperMag's 2.5 ms budget. The skin code is computed once per calibration, so conditioning adds almost nothing per reading.

The analytical core is a drift decomposition carried out before any architecture comparison. For each new skin or condition, we fit the best per-channel affine map from its raw readings to the reference skin's readings on identical presses. We then measure how much reconstruction error is left after that correction when a model trained on the reference skin is used. If the affine-corrected residual is close to in-distribution error, drift is input-affine, and conditioning inside the network cannot be justified. If the residual stays large, and grows with wear and temperature, the claim is supported and we can measure how much of the remaining error comes from the skip pathway.

Baselines

The naive reference points are bicubic interpolation of the magnetometer grid and a plain U-Net trained on the reference skins and applied unchanged to new conditions. SuperMag reports that interpolation is clearly worse than learned reconstruction but gives no numeric PSNR for interpolation in the file. The plain U-Net sets the size of the lab-to-field gap we are trying to close.

The strongest existing method is SuperMag's conditional variational autoencoder, which reports 24.24 ± 2.88 dB PSNR and 0.78 ± 0.06 SSIM on its own Allen-key and letter data. It matters because it has no skip connections at all, so it may be naturally less exposed to drift leaking through skips. We retrain it on our data with the same training skins, the same tuning budget in trials, and the same latency ceiling. To separate the effect of the data from the effect of the architecture, we also give it the skin code as an extra condition. Beating it means higher PSNR, SSIM and edge accuracy on held-out skins, worn skins and warm skins, while at least matching it on the training-condition split.

Three recalibration baselines use exactly the same nine presses as our method. The first re-zeroes each channel against a no-contact reading. The second applies the per-channel affine correction from the decomposition experiment. The third fine-tunes the plain U-Net for a fixed number of steps on the nine presses. A fourth baseline follows ReSkin's triplet-loss adaptation, given several hundred unlabeled pokes. This is deliberately more field data than our method uses, and we report it as such.

The ablations identify which ingredient matters. The first applies conditioning only at the bottleneck, as SITR and CTSRL effectively do, rather than at the skips. The second removes the skips entirely while keeping conditioning. The third replaces the press-derived code with a one-hot skin identity, in the style of CTSRL, which cannot generalize to a skin never seen in training. The mechanism is supported only if skip-level conditioning beats bottleneck-only conditioning on the shifted splits without losing edge accuracy in distribution.

Why not just…?

Why not just re-zero the sensor and rescale each channel? This is the most serious threat, and it is cheap. If magnetization differences between cast skins mostly produce offsets and gains, the affine baseline will close most of the gap. The decomposition experiment tests this directly, and we will report the outcome whichever way it falls. We expect it to fall short for two physical reasons. A replaced skin can sit slightly shifted relative to the magnetometers, which changes the spatial pattern of the field and not just its scale. Temperature changes elastomer stiffness, which alters the nonlinear relation between deformation and field. That expectation is untested, and the proposal is designed so that it could be proven wrong.

Why not just keep SuperMag's bottleneck model, which has no skips to leak through? Removing the skips may indeed make the model more robust. SuperMag, however, reports trouble with sharp edges and micro-texture, which is the fidelity skips are meant to restore. The conditioned-SuperMag and no-skip ablations show whether we are trading robustness for sharpness or getting both.

Why not just fine-tune on a few presses from the new skin? Fine-tuning on nine presses risks overfitting and requires an optimization loop on the robot controller. Conditioning requires only one forward pass. Both use the same presses, so the comparison is direct. If fine-tuning matches us, the contribution shrinks to a deployment convenience, and we would say so.

Why not a transformer with calibration tokens, as in SITR? SITR needs a million synthetic images, and no photorealistic simulator for magnetic skins exists in the file. The file also flags that invariance objectives of this kind wash out fine detail. The bottleneck-only conditioning ablation serves as the small-data version of this approach.

Proposed Experiments

The main result is a grid of models against splits. The models are the baselines and the proposed network. The splits are training skins, held-out cast skins, skins after each wear block, and each temperature, and the Allen-key and letter indenters follow SuperMag. Metrics are PSNR, SSIM, and an edge F-score computed on thresholded depth-map boundaries, which targets the sharp-detail weakness reported for SuperMag. The key summary number is the gap ratio: fidelity on a shifted split divided by in-distribution fidelity.

The downstream check repeats SuperMag's in-hand angle-correction task on a swapped skin with no retraining. SuperMag reports a 100% success rate against 16.7% for its baseline, but only on the training skin. Latency is measured on the robot's controller hardware against the 2.5 ms budget.

The decisive outcomes are narrow. The idea succeeds if three things hold. The affine-corrected residual stays large on at least the wear and temperature splits. Skip-level conditioning beats both SuperMag and bottleneck-only conditioning on those splits. And it does not lose edge F-score in distribution. It fails, and should be dropped in favour of input recalibration, if the affine baseline comes within noise of the proposed model everywhere.