When a robot fingertip starts to slip, or when a human finger slides across a film of water rather than thickened syrup, the useful information is in how the contact changes over time. The surface catches and releases, relaxes, and oscillates. Current tactile learning either feeds these signals to large neural networks trained on millions of frames, or summarizes them with hand-built features fed to a recurrent network. Neither says what the dynamics are: how fast the contact relaxes, and at what rhythm it sticks and slips.
We propose to take the deformation field that a camera-based tactile sensor sees and fit a single linear map that advances a nonlinear embedding of that field by one time step. The sensor is a camera watching a soft gel with printed markers, and the field is how those markers move. The embedding uses random Fourier features, which are fixed random cosine projections that approximate a Gaussian kernel. The eigenvalues of the fitted map can be read directly as physical quantities: decay rates correspond to relaxation times, and rotation angles correspond to stick-slip frequencies. The map can also be run forward to predict slip before it happens. The claim is that this small, interpretable descriptor, fitted from a few thousand frames, predicts translational and torsional slip severity and separates lubrication regimes as well as or better than recurrent networks and pretrained tactile encoders, while exposing timescales that can be checked against rheometer measurements.
Core Claim
Let $f_t \in \mathbb{R}^d$ be the per-frame physics features that Learned Slip-Detection-Severity Framework extracts from GelSight marker flow: net velocity, divergence, curl, and contact area, computed over a short window of past frames. We lift these features with $z_t = \sqrt{2/r}\cos(W f_t + b)$, where $W$ is an $r \times d$ matrix of Gaussian random frequencies and $b$ is a vector of random phases. We then fit a linear operator $K$ by ridge regression:
$$ K = \arg\min_K \sum_t \lVert z_{t+1} - K z_t \rVert^2 + \lambda \lVert K \rVert_F^2 . $$
Each eigenvalue of $K$ can be written as $\mu_i = \exp\big((-1/\tau_i + i\,\omega_i)\,\Delta t\big)$, where $\Delta t$ is the frame interval. Here $\tau_i$ is a relaxation time and $\omega_i$ is an angular frequency. In this reading, stick-slip shows up as a lightly damped, rotating eigenvalue pair. Viscoelastic creep, whose effective modulus changes over orders of magnitude within the first two seconds of touch (Establishing direct relationships between soft material perception and rheology), shows up as a spread of real eigenvalues across timescales.
The method comes from two places. From KDP we take the observation that curved trajectories become nearly linear in random-Fourier-feature space and that the resulting operator can be calibrated from about 4,000 samples. From Invertible Kernel PCA with Random Fourier Features we take a way to map lifted states back to physical units.
Three things are new relative to those papers. First, we apply the idea to physical contact streams rather than transformer hidden states. Second, we fit operators both globally and per contact episode, and treat the spectrum itself as the perceptual descriptor. Third, we test the spectrum against independently measured physics.
A skeptic will note that this is extended dynamic mode decomposition, a known family of lifting methods, with a random kernel dictionary. We agree. The contribution is not the estimator. It is the question of whether contact dynamics admit a compact linear-in-lift description whose spectrum carries the tactile information that How to identify water from thickener aqueous solutions by touch and Tapered whiskers are required for active tactile sensation attribute to stick-slip transients.
Research Program
The work has three parts.
Build the lifted operator pipeline. We use the four Jawale field quantities plus a short delay embedding of them, random-feature dimensions $r$ from 50 to 2,000, and both a ridge decoder and the invertible kernel PCA decoder to recover predicted slip velocity. The inversion paper reports that large regularization over-smooths sharp transients and small regularization collapses toward linear PCA. That result constrains us, so we sweep $\lambda$ and report reconstruction error separately on burst frames and on quiescent frames.
Collect two small datasets that the file's resources lack. The first follows the Jawale protocol: GelSight Mini, ArUco ground truth, 15 training objects and 5 held-out objects. We extend it with in-hand rotational slip, which Jawale leaves open and which Sparsh says its pretraining data under-covers. The second has the sensor slide at controlled speed and normal load over a rigid plate coated with water, glycerol dilutions, and xanthan-thickened solutions. Each fluid's viscosity is measured on a rheometer.
The camera frame rate is an honest constraint. At roughly 25–30 Hz, the GelSight camera cannot resolve the high-frequency bursts described in the water-identification study. We therefore mount a MEMS accelerometer on the sensor and choose low sliding speeds where stick-slip periods fall within the camera's Nyquist limit, the highest frequency a sampled signal can represent. The operator is fitted on camera features alone, on the accelerometer alone, and on both.
Analyze the spectrum as a physical measurement. For each lubrication episode we fit a per-episode $K$ and extract its dominant $\tau_i$ and $\omega_i$. We then ask whether these track measured viscosity and sliding speed monotonically. As a secondary analysis, and only if the authors share their trajectories, we lift Burgeson's force–indentation time series and test whether fewer kernel components than their six linear principal components (which explain 91% of variance) predict participants' two-alternative forced-choice firmness judgments.
Baselines
Naive reference points. The first is persistence: predict the next slip velocity as the current one, and the lifted state as unchanged. The second is a class-prior predictor for lubrication labels. Any method that fails to beat persistence at short horizons is only reproducing smoothness.
Strongest existing method on this question. This is the LSTM of Learned Slip-Detection-Severity Framework. On the same four field features it reports 0.26 cm/s mean absolute error for slip velocity in cross-validation and 92% slip detection on unseen objects. It is the strongest competitor because it uses exactly our inputs, so any gain must come from the dynamics model rather than from the features.
We retrain it on our re-collected data using the same object splits, the same input window, and a hyperparameter budget equal to ours (the same number of configurations searched on the same validation objects). We also train it with 10%, 30%, and 100% of the training episodes. We count as beating it either of the following:
- matching 0.26 cm/s (or whatever our reproduction gives) using at most 30% of the episodes;
- lower error on held-out objects and on the new torsional episodes at equal data.
Strongest pretrained-representation competitor. This is a frozen Sparsh encoder with a linear or small recurrent head trained on the same labels. Sparsh reports an average 95.1% improvement over end-to-end models on TacBench in the 1–10% labeled regime, so it is the natural low-data rival. AnyTouch 2's frame-difference objective is built for transients, but we have no reported slip or lubrication number for it in the file, so we include it only if weights are available.
Ablations that isolate the key ingredient. The first is linear dynamic mode decomposition on the raw features, with no lift. It tests whether the nonlinear embedding matters. The second is random Fourier features with a direct ridge readout and no operator. It tests whether the temporal operator matters or only the kernel. The third compares a global operator against per-episode operators, to test whether the spectrum descriptor adds information.
Why not just…?
Why not just train a small recurrent network on the same few thousand frames? This is the main threat, since Jawale's LSTM already performs well. Our argument is not raw accuracy at full data. We make two narrower claims. First, a ridge-fitted operator has no stochastic training and degrades more gracefully as episodes are removed. Second, its spectrum gives timescales that the LSTM does not expose. The data-fraction sweep settles the first claim. If the LSTM matches us at 10% data and on torsional slip, the predictive claim fails and only the interpretability claim remains.
Why not just take the Fourier spectrum or wavelet features of the divergence and curl signals? This is a real threat for the lubrication task, because stick-slip is oscillatory. It is weaker where regimes change mid-episode: frequency transforms assume a stationary signal and have no way to represent a transition from sticking to slipping or a decaying relaxation. We include spectrogram and wavelet features with gradient boosting as a baseline. If they match the operator spectrum on classification and viscosity correlation, we will report that the lift is unnecessary for this task.
Why not just use a pretrained tactile foundation model and a linear probe? Sparsh's latent objectives are designed to ignore high-frequency pixel variation. That is exactly what carries stick-slip. AnyTouch 2 had to add a frame-difference loss to keep such transients. The frozen-Sparsh baseline on the lubrication data tests directly whether this matters in practice.
Proposed Experiments
We run four experiments.
Slip prediction. We report velocity error at horizons of one, five, and ten frames, on held-out objects, separately for translational and torsional slip. We also measure lead time: how many frames before ArUco-detected slip onset each method's predicted velocity crosses a threshold.
Lubrication. On the rheometer-measured fluids we report classification accuracy with 5, 20, and 100 episodes per fluid. We also report the Spearman rank correlation between the dominant fitted $\tau_i$ or $\omega_i$ and measured viscosity at fixed sliding speed.
Sensitivity sweeps. We sweep random-feature dimension and ridge strength, reporting error on burst frames and quiescent frames separately. This tests whether the lift preserves transients or smooths them, as the inversion paper warns.
Decisive check. The proposal succeeds if two things hold. The lifted operator must beat both linear dynamic mode decomposition and the Fourier-feature baseline on regime-changing episodes. Its fitted timescales must also rank fluids by viscosity with $\rho > 0.8$, which none of the neural baselines provide. If linear dynamic mode decomposition does as well, the contact dynamics are effectively linear at these scales, and the nonlinear mapping direction adds nothing to this problem. That would also be worth reporting.
References
- Learned Slip-Detection-Severity Framework using Tactile Deformation Field Feedback for Robotic Manipulation
- Establishing direct relationships between soft material perception and rheology
- KDP: Simplifying Representation Dynamics in Kernel Space
- Invertible Kernel PCA With Random Fourier Features
- How to identify water from thickener aqueous solutions by touch
- Tapered whiskers are required for active tactile sensation
- Sparsh: Self-supervised touch representations for vision-based tactile sensing
- AnyTouch 2: General Optical Tactile Representation Learning For Dynamic Tactile Perception