Robots with camera-based touch sensors, which press a soft gel against an object and photograph how it deforms, can be trained cheaply in simulation to tell whether they are touching a flat face, a curved face, an edge, or a corner. Classifiers trained on simulated gel images fail badly on real ones, because simulated and real images differ in lighting, reflection, and gel texture. The current best fix, Sim2Surf, stacks two heavy adaptation stages. The first is a diffusion model, a generator that learns to turn noise into images, used here to restyle simulated images so they look real. The second is a domain-adversarial network that forces features from the two domains to look alike. Together they raise real-world accuracy from 34.7% to 81.9%. Nobody knows which stage does the work. Nobody has asked whether the classical geometric description of a surface, the two principal curvatures that say how sharply it bends in its most- and least-bent directions, would make most of that machinery unnecessary.
We propose to settle this with a controlled study. We route classification through an explicit geometric intermediate: predict a height map from the tactile image, compute principal curvatures from it, and classify the resulting shape-type and bending-strength maps. We then cross this representation choice with every combination of Sim2Surf's adaptation stages, run on the same data and with the same tuning budget. The underlying bet is that the sim-to-real gap lives mostly in appearance, while geometry is the same in both domains. Simulators give exact depth for free, so a model forced to pass through geometry should inherit the simulator's one honest signal. The study is also an empirical test of a textbook invariance claim that has never been checked on noisy physical touch data.
Core Claim
Sim2Surf uses curvature only to generate labels from CAD meshes, through covariance singular values. Its classifier still reads raw image appearance. We claim that making curvature the input to the classifier recovers most of Sim2Surf's gain at a fraction of the cost, and that it degrades more gracefully at light contact force.
The second part of the claim is a mechanistic prediction. The shape index $S = \frac{2}{\pi}\arctan\frac{k_2+k_1}{k_2-k_1}$, where $k_1 \ge k_2$ are the principal curvatures, depends only on the ratio of the two curvatures. Uniformly scaling the indentation depth therefore leaves it unchanged. The curvedness $C=\sqrt{(k_1^2+k_2^2)/2}$ scales with depth. If this holds on real gels, distinctions that depend on shape type should survive weak presses: corners are caps, edges and curved faces are ridges. Distinctions that depend on bending magnitude should not: flat versus curved, curved versus edge. Those should collapse as force drops.
This decomposition explains the force failure that Sim2Surf reports only qualitatively, and it can be falsified directly. It also echoes Qualitative Structure From Motion, which found that the sign of Gaussian curvature can be recovered under noise where exact metric reconstruction fails. Our version tests the same "qualitative survives, quantitative doesn't" idea on dense real sensor data rather than synthetic three-point stimuli.
Research Program
The geometric path has three stages. First, a U-Net predicts a height map from each tactile image. It is trained on simulated pairs, where the TACTO simulator supplies exact depth. Second, we fit local quadratics at two or three neighborhood scales and take the eigenvalues of the fitted Hessian to obtain $k_1, k_2$. From these we compute $S$ and $C$ maps. Third, a small convolutional classifier labels the contact.
We also build a first-order variant that stops at surface normals, following the image-to-normal approach of PatchGraph. PatchGraph showed such normals transfer to real DIGIT images well enough for tracking within 4 mm, but it never computed second-order quantities.
The height predictor faces its own sim-to-real gap. We therefore treat it as a component to ablate, not an assumption. The diffusion restyling and the adversarial alignment can each be applied to the height predictor rather than to the classifier.
The methodological backbone follows the prescriptions in Troubling Trends in Machine Learning Scholarship: compound systems must isolate where their gains come from. It also follows Winner's Curse? On Pace, Progress, and Empirical Rigor: baselines get equal tuning budgets, and results are sliced by data subgroup.
Those papers supply the method. Sim2Surf supplies the gap. The cautionary results from time-series forecasting constrain our expectations: in Are Language Models Actually Useful for Time Series Forecasting?, elaborate modules were matched by simple preprocessing. We explicitly allow the same outcome here, with color jitter or histogram matching matching diffusion, and design the study to detect it.
Baselines
Do-nothing reference. The do-nothing reference is a classifier trained on raw simulated images and tested on real ones. Sim2Surf reports 34.7% accuracy and 0.0% corner F1 for this setting. We reproduce it to check that our pipeline matches theirs.
Blindfold baseline. We add a degenerate "blindfold" baseline in the spirit of Blindfold Baselines for Embodied QA, where a question-only model reached 50.34% and beat full agents. This classifier sees only the binary contact-area silhouette and total contact intensity, not the gel's shading. The silhouette's outline may already reveal edges and corners. If this baseline is close to 81.9%, curvature is beside the point for most classes.
Strongest existing method. The strongest competitor is the full Sim2Surf pipeline: a frozen DINOv2 backbone, diffusion restyling, and adversarial alignment. It reports 81.9% accuracy and 68.3% corner F1 on its 792-image real test set. Its tracking result is 2.22 cm position error and 86.75% ADI-AUC, against 4.21 cm and 75.68% for the geometric tracking baseline.
We run it from released code and data where available. Otherwise we re-implement it on the same 50,000 simulated images from 10 YCB objects and the same 5,000 unlabeled real images.
Fairness and the bar to beat. Every arm gets the same number of hyperparameter trials and the same GPU-hour cap, and all arms are evaluated over five seeds. With 792 test images, the 95% interval around 80% accuracy is roughly ±2.8 points. "Beating" Sim2Surf therefore means one of two things. The first is matching its accuracy within that interval without diffusion, at a stated reduction in training and inference cost. The second is exceeding it by more than the interval on the lowest-force third of contacts.
Key ablation. The ablation that isolates our ingredient is the representation factor. We compare raw image, first-order normals, and second-order curvature maps under each adaptation setting. Within the curvature arm, we further compare shape index alone, curvedness alone, and both, plus Sim2Surf's covariance-eigenvalue heuristic in place of Hessian curvatures.
Why not just…?
Why not just tune the adversarial alignment, or use color jitter, instead of diffusion? This is a real threat, and the factorial design addresses it directly rather than dismissing it. Sim2Surf reports only the endpoint combination, so no evidence in the literature shows diffusion is needed. Our grid includes adversarial alignment alone, diffusion alone, and cheap photometric augmentation alone, each tuned with the same budget. If jitter plus alignment on raw images reaches about 80%, we report that as a primary finding. In that case the geometric bottleneck's case rests on the force-sliced results.
Why not let a convolutional network learn curvature implicitly from pixels? Mesh-MLP shows unstructured networks match specialized geometric operators on clean meshes, scoring 100% on SHREC11 and 90.9% on Manifold40. Those benchmarks have no domain shift, though. On tactile data, the implicit route is exactly the 34.7% raw-simulation result. The comparison between the raw-image and curvature arms, run under identical adaptation, measures whether explicit geometry adds anything once appearance shift is handled.
Why not stop at surface normals? Normals are easier to estimate and suffice for PatchGraph's tracking. PatchGraph itself, however, concedes that flat and uniformly curved patches leave motion unobservable. Classical work by Gurfinkel et al. needed second differences of pressure to separate spheres, cylinders, and planes. The normals-versus-curvature arm tests this on modern sensors. If normals tie with curvature, second-order estimation is not worth its noise, and we will say so.
Proposed Experiments
The main experiment is the full grid. It crosses three representations with four adaptation settings (none, diffusion, alignment, both), and adds the jitter and blindfold arms. All arms are trained on the Sim2Surf simulated and unlabeled real sets and scored on its real test set. Metrics are accuracy, per-class F1 with special attention to corners, wall-clock training and inference cost, and downstream pose-tracking error using Sim2Surf's tracking protocol.
Decisive check: force sweep. The decisive check uses new data. We collect real contacts on the same YCB objects with the robot arm at five controlled normal-force levels, measured by a wrist force sensor, and re-score every arm on each force level.
The invariance hypothesis predicts a specific pattern for the curvature arm. Corner and edge-versus-cap discrimination should stay roughly flat across force. Flat-versus-curved confusion should rise as force falls, and the raw-image arms should degrade uniformly. If the shape-index-only classifier shows no flatter force curve than raw images, the invariance claim fails on real gels. If the blindfold arm tracks the full model at high force but not low, contact silhouette is doing the work there.
Secondary transfer test. We repeat the representation comparison, without retraining the adaptation stages, on the real data from Tactile MNIST. That set contains 153,600 GelSight Mini samples, a different sensor. We use it for per-touch local surface-type labels derived from its 3D-printed digit meshes. No reported per-touch classification result was found on Lacuna for this dataset, so here we report relative differences between arms only.
References
- Sim2Surf: A Sim2Real Surface Classifier for Vision-Based Tactile Sensors With a Bilevel Adaptation Pipeline
- Qualitative Structure From Motion
- PatchGraph: In-hand tactile tracking with learned surface normals
- Troubling Trends in Machine Learning Scholarship
- Winner's Curse? On Pace, Progress, and Empirical Rigor
- Are Language Models Actually Useful for Time Series Forecasting?
- Blindfold Baselines for Embodied QA
- Mesh-MLP: An all-MLP Architecture for Mesh Classification and Semantic Segmentation
- Some Aspects Of Sensory Instrumentation For Robots And Manipulators
- Tactile MNIST: Benchmarking Active Tactile Perception