When a robot grips a towel, a sponge or a cable, the part it most needs to track is the part it can least see. That region sits inside the fingers, hidden from every camera. It also stops obeying the rule most trackers are built on: that an object is a rigid solid whose whole state is a position and an orientation. Touch sensors help only partly. A fingertip sensor can report that it is pressing on something and roughly what shape that something has, but on a plain, textureless, deforming surface it cannot report which point of the object it is touching. We propose to supply that missing fact with cheap passive radio tags sewn or laminated into the object at known locations, read by small magnetic coils in the robot's fingertips. Each successful read is a short-range, identity-bearing event: "material point 17 is within a few millimetres of the left finger right now." Cameras need line of sight to report this; the radio read does not.
The idea borrows from physical-to-digital bookmarking, the long tradition of attaching unique machine-readable identities to physical things so a system can look them up. That tradition has always treated a tagged item as a dimensionless database key. We instead use tag identity as a geometric constraint on a deforming surface. The same constraints give a direct, measurable test of when the rigid-object assumption has broken inside the grasp.
Core Claim
Sparse, binary, identity-bearing proximity reads from fingertip near-field coils are enough to keep a non-rigid surface tracker locked onto the grasped region through full visual occlusion. Continuous RF localization is not needed. Only "tag i is inside finger j's small read zone, yes or no" is needed. Near-field RFID has already shown it can deliver exactly this kind of spatially confined presence read reliably. The claim is useful because it sidesteps the strongest objection to RFID in manipulation, which is its coarse spatial resolution. We never ask the radio for a position, only for a name at a place the robot already knows: its own fingertip.
Where This Sits in Prior Work
Two results motivate the proposal. TEG-Track fuses RGB-D vision with GelSight touch sensors, which image the deformation of a soft gel skin, to track objects held in the hand. It classifies contact as sticking or slipping and converts that into velocity constraints on a 6-degree-of-freedom rigid pose. It reports a 30.9% drop in rotation error and 21.4% in translation error over BundleTrack, a model-free rigid tracker that optimizes a pose graph over past keyframes. TEG-Track's authors name non-rigid objects as the open case. Its rigid solver has no way to express surface strain, so it must misread strain as slip. Textureless Deformable Object Tracking With Invisible Markers shows the other half of the problem. It paints UV-fluorescent dots that ordinary cameras cannot see and uses them to build the DOT benchmark. Its paper sequence shows markerless tracking failing within 20 frames while marker-based tracking survives the whole sequence. Those markers are optical, though, so they disappear the moment cloth folds or a gripper closes over them. We reuse that paper's hardware differently: it is our ground-truth rig, not our method.
Three results constrain the design. RFID+ pairs ordinary long-range UHF RFID with controlled near-field magnetic coils. Its main trick is repurposing the small matching loop already present on commercial tags so they can be read by a magnetic field confined to a chosen region. In a textile warehouse this cut miss-reads from 22.9% to 1.06% and eliminated reads from outside the target region. That is the capability we need, but only at warehouse scale: RFID+ reports no read-zone size at fingertip scale, and no behaviour under tag bending. We therefore treat both as quantities to measure, not assume. Enabling Robot Manipulation of Soft and Rigid Objects shows that DIGIT fingertip sensors confuse deformation of their own gel with slip or object deformation. Its fix is a threshold that doubles after each slip. That is a warning that touch alone cannot disambiguate the grasped geometry. GarmentLab documents how occlusion and dynamics mismatch wreck visual garment policies. Its best real T-shirt folding result is 10/15, obtained with keypoint embedding alignment.
For the tracker itself we take the non-rigid registration setting of Grasp Transfer for Deformable Objects by Functional Map Correspondence. That work transfers grasps across deformed meshes offline, and it reports that classical registration collapses: ICP reaches 31.25% and CPD 56.25% accuracy on its transfer test. We work online, from partial sensor data, and add anchors that neither registration method has.
Research Program
The work has three parts, and the first can kill the project.
Part 1: characterise the sensor. We build fingertip-scale versions of the segmented-coil design from RFID+ and mount them behind GelSight-style sensors on a parallel gripper. We then measure the effective read radius $r$. We also measure its miss-read and false-read rates as functions of tag orientation, standoff, fabric and foam substrates, and bending curvature. This directly measures the gap that bookmarking work leaves open: how strain on a flexible substrate detunes a tag. If bending shrinks or distorts the read zone, the calibration curve itself becomes a coarse curvature signal. We will report it either way.
Part 2: build the tracker. Each object gets a template mesh with $K$ tags at known surface locations $u_i$. The surface is modelled as an embedded deformation graph, a sparse set of control nodes whose local rotations and translations warp the mesh, with parameters $\theta_t$ at time $t$. The deformed position of tag $i$ is $x_i(\theta_t)$, and fingertip coil $j$ sits at $f_j(t)$, known from the robot's kinematics. We minimise
$$ E(\theta_t) = E_{\text{depth}} + E_{\text{touch}} + E_{\text{reg}} + \lambda \sum_{(i,j) \in R_t} w_{ij}\,\max\!\big(0,\ \lVert x_i(\theta_t) - f_j(t)\rVert - r\big)^2 . $$
Here $E_{\text{depth}}$ aligns the visible surface to camera depth, $E_{\text{touch}}$ aligns contact patches from the touch sensors, and $E_{\text{reg}}$ keeps neighbouring nodes moving similarly. $R_t$ is the set of tag–finger reads at time $t$, and $w_{ij}$ is a confidence weight set from the Part 1 false-read rate. A weaker penalty for absent reads during confirmed contact is weighted by the measured miss rate.
Part 3: test for rigidity. We fit the best single rigid transform to all currently anchored tags. A residual well above $r$ is direct evidence that the grasped region is deforming. This gives a principled switch: stay in TEG-Track's rigid mode for objects that behave rigidly, and hand over to the deformation graph when they do not. This replaces the threshold-doubling heuristic of Bambach et al. with a geometric test.
Hardware and data. Tags are commercial UHF inlays on flexible substrates, sewn into cloth, laminated under foam, or sleeved along cables. We collect our own sequences because no existing benchmark contains RF reads. There are four object groups: towels and T-shirts, sponges and foam blocks, cables, and rigid control objects from TEG-Track's categories. Ground truth comes from the DOT-style UV marker rig. Its markers are invisible under normal light, so they give dense surface ground truth without leaking texture to any vision baseline. Multi-view capture keeps the grasped region partly observable to ground truth even when the robot's own cameras cannot see it.
Baselines
The naive reference is visual-only non-rigid tracking: the same deformation graph driven by depth alone, plus a CPD registration variant. De Farias et al. report CPD at 56.25% on their transfer accuracy test and ICP at 31.25%, and DOT reports markerless tracking failing within 20 frames on paper. We expect these baselines to hold while the surface is visible and drift once the gripper closes.
The strongest existing method is TEG-Track run on the same sequences with the same cameras and GelSight sensors, initialised identically. Its reported result is a 30.9% reduction in average rotation error and 21.4% in translation error relative to BundleTrack on its real in-hand dataset. On deformable objects there is no reported result found on Lacuna for TEG-Track or any visuotactile tracker, which is exactly the gap. To compare fairly we score TEG-Track's rigid pose by warping the template rigidly and measuring per-vertex error. We also give it a fairer non-rigid companion: our deformation graph with depth and touch terms but no tag term, tuned with the same hyperparameter-search budget as the full system.
We add a visible-marker oracle: the deformation graph fed the UV markers as if the robot's own cameras could see them. This bounds what optical bookmarks could give under the robot's actual viewpoints.
Two ablations isolate the key ingredient. In the first, tags are anonymised: reads report "some tag is here" but not which one, so they reduce to extra contact evidence. The gap between anonymous and named reads measures the value of identity itself. In the second, the near-field coils are replaced by ordinary far-field reads, testing whether spatial confinement is necessary.
The full system beats the field if it clears three bars. First, it must lower grasped-region per-vertex error and lengthen tracking survival relative to the depth-plus-touch deformation graph, in the occluded frames specifically. Second, it must match TEG-Track within noise on rigid controls. Third, it must beat the anonymised-tag ablation, since otherwise identity adds nothing.
Why not just…?
Why not just use visual foundation-model correspondence or more cameras? On the visible surface this is a real competitor, and we expect it to be as good as our method there. The proposal makes no claim about visible regions. The evidence against vision in the grasped region is geometric: the region is occluded by the fingers. DOT's 20-frame failure shows how quickly textureless surfaces are lost even when visible. Our split of error by occlusion state settles the question. If a wrist camera plus the multi-view rig leaves the grasped region visible most of the time, the tags add little, and we will report that fraction first.
Why not just print or emboss fiducial codes that the fingertip sensor can read? This is the most serious alternative, because a GelSight sensor does image the contact imprint. But it can read a code only when the code is pressed flat and fully inside the contact patch. On folded cloth or cables, contact patches are partial and sheared. Embossing also changes the surface the robot is trying to handle. We include an embossed-code variant on the towel and foam objects and compare identity-recovery rate per contact event. If embossed codes match the RF reads, the better design is the cheaper one, and we would say so.
Why not just rely on touch heuristics, or simulate the object instead? Bambach et al. show that threshold heuristics keep fragile objects intact but cannot recover shape or support goal-directed deformation. They needed 11 slip events and 45% compression on a hard connector. Simulation needs a measured state to start from, and GarmentLab and DaXBench both document persistent cloth reality gaps. Our tracker could supply that state, so it complements simulation rather than competes with it.
Proposed Experiments
The first decisive check is Part 1. If the fingertip read radius exceeds roughly a centimetre, or bending detunes tags into frequent misses, the anchors are too loose and the project stops, or moves to smaller coils. The main experiment runs in-hand regrasp, sliding and folding sequences on all four object groups. It reports per-vertex error on grasped and visible regions separately, the length of time before error exceeds a fixed threshold, and the accuracy with which contacted template points are identified. The rigidity test is scored against ground-truth strain as a detector of deformation onset. On rigid controls we check that switching to rigid mode reproduces TEG-Track's accuracy. Finally, for a downstream check, we feed tracked tag locations to a keypoint-based T-shirt folding policy on real garments. GarmentLab's 10/15 folding success with keypoint alignment is the reference point, and the question is whether in-grasp anchors recover the failures caused by occlusion.
References
- Enhancing Generalizable 6D Pose Tracking of an In-Hand Object with Tactile Sensing
- BundleTrack: 6D Pose Tracking for Novel Objects without Instance or Category-Level 3D Models
- Textureless Deformable Object Tracking With Invisible Markers
- RFID+: Spatially Controllable Identification of UHF RFIDs via Controlled Magnetic Fields
- Enabling Robot Manipulation of Soft and Rigid Objects with Vision-based Tactile Sensors
- GarmentLab: A Unified Simulation and Benchmark for Garment Manipulation
- Grasp Transfer for Deformable Objects by Functional Map Correspondence
- DaxBench: Benchmarking Deformable Object Manipulation with Differentiable Physics