Speed-Equivariant Texture Vibration from a Generated Height Map

When a fingertip or probe slides across fabric or brushed metal, the vibration it picks up is mostly the surface's height profile read out over time. A surface feature with spatial period $\lambda$ passed at speed $v$ produces vibration at roughly $v/\lambda$. This proposal uses that relation to tie two signals together. It generates a micro-scale height map from an ordinary photo with a diffusion model, which is a generator that learns by gradually turning random noise into realistic samples. It then predicts the friction vibration for any sliding speed by mapping the height map's spatial spectrum onto temporal frequency and passing it through a small learned contact filter. The contact filter models how the finger or probe smooths fine features.

The payoff is practical and testable. Current visual-to-tactile generation, most directly HapticMatch, produces height maps from photos and records vibration audio at several speeds and pressures, but never generates the two together. A model that predicts vibration directly from an image has no reason to behave sensibly at sliding speeds it never saw. The physics-structured route should extrapolate to new speeds and arbitrary speed profiles. It also guarantees that the rendered geometry and the rendered vibration describe the same surface.

Core Claim

The vibration power spectrum of a texture under steady sliding is well approximated by the generated surface's spatial spectrum, rescaled by speed and shaped by a contact transfer function that depends only weakly on the material:

$$ S_a(f; v, p) \approx \frac{1}{v}\,\bigl|H_\theta(f; v, p)\bigr|^2\, S_h\!\left(\tfrac{f}{v}\right) $$

The symbols are as follows. $S_a$ is the vibration power spectral density at temporal frequency $f$ for sliding speed $v$ and contact pressure $p$. $S_h(k)$ is the one-dimensional spatial power spectrum of the height map along the sliding direction at spatial frequency $k$. $H_\theta$ is a small learned filter for the contact patch and sensor response. The $1/v$ factor comes from changing variables from space to time.

The claim makes two predictions. First, the spectrum $S_h$ must come from a sampler that preserves high-spatial-frequency power, so a diffusion prior should beat deterministic height regression on vibration prediction, not just on height realism. Second, this structure should beat direct image-to-vibration predictors when tested at held-out speeds. For variable-speed strokes, the same model renders a time signal by sampling the generated profile along the position trajectory $x(t)$. We represent that trajectory with a few low-frequency discrete cosine transform (DCT) coefficients, following trajectory-space motion models Learning Progressive Joint Propagation. The motion stays smooth and low-band, and the texture supplies the high band.

Relation to Prior Work

HapticMatch supplies the dataset, the height generator, and the evaluation metric. Its rectified-flow transformer (RF-DiT-XL-2) translates RGB photos into GelSight height maps. It reports lower 2D power-spectrum error than Pix2Pix, and its vibration recordings are left unused by the generator. We reuse its generator as the source of $S_h$ and add the missing vibration output. We do not train a new image model.

Three lines of work model signals in the frequency domain, and each motivates or constrains one part of the design:

  • DiffusionPhase runs diffusion over periodic phase parameters of human motion. It gains length-robustness but struggles with exact cycle counts. That weakness is a warning: a purely spectral vibration model may get timbre right while losing exact event timing. This is why our time-domain rendering samples an explicit profile rather than a free spectrum.
  • Fourier Controller Networks truncate high frequencies on the premise that physical motion is low-band. Applied naively to a probe scanning a texture, this would erase exactly the roughness we want. Our formulation separates the bands explicitly: the stroke kinematics are low-band, and the vibration is the high band generated by the surface.
  • Beyond Pixel Space shows that diffusion uncertainty concentrates in low DCT frequencies. It also notes that the DCT's global support causes leakage. We use spectra only as summary targets and evaluation, so this leakage does not enter generation.

TacDiffusion shows that diffusion outputs at 50–150 Hz need a hand-designed second-order filter to drive 1000 Hz contact control. Our render-from-profile step produces vibration at any sample rate with no interpolation filter. We note this as a side benefit, not a central claim.

The argument for diffusion over regression on the height step comes from the regression-to-the-mean analyses in He et al. and Almansa et al.. Averaged height maps lose high-frequency power. Under our mapping, that missing power becomes missing high-frequency vibration, so this deficit should show up directly in the vibration metrics.

Research Program

Is the law true in the data? Before building anything, we check whether the core relation holds at all. For each of HapticMatch's 100 materials, we estimate how much the vibration spectrum shifts across its recorded speeds. We fit a frequency-scaling exponent, which is 1 under the claim, and record the residual. We expect periodic textures such as woven fabrics and machined metals to obey the scaling. Stick-slip materials such as rubbery plastics may not, because their vibration depends on friction dynamics rather than geometry. We will report the split honestly and restrict the strong claim to the materials where the law holds.

Building the model. The pipeline has four parts:

  1. Draw several height samples per photo from the HapticMatch rectified-flow model.
  2. Compute the directional spatial spectrum $S_h$, averaging over samples and over line profiles. This averaging is appropriate because the target is a power spectrum, not a single realization.
  3. Fit $H_\theta$, a small network conditioned on speed, pressure, and a coarse material embedding, by minimizing log-spectral distance to the recorded vibration.
  4. Handle frequency coverage. The GelSight map's pixel pitch caps the usable spatial frequency at $k_{max}$, so the model is only defined below $v\,k_{max}$. Above that band, $H_\theta$ is allowed a separate residual term, which we report separately so the physics-explained portion is visible.

Controlled recordings with known speed profiles. Speed and pressure labels in HapticMatch are coarse. To get clean labels, we will collect a controlled set: a motorized linear stage with encoder-logged speed profiles and a GelSight-mini scan of each sample before sliding. Following the logic of the CMM-guided acquisition in Farman Institute 3D Point Sets, mechanical positioning rather than vision provides the kinematic ground truth. An accelerometer on the probe records vibration. This set adds variable-speed strokes, encoded as DCT coefficients, to test time-domain rendering.

Baselines

Do-nothing reference. Return the mean training vibration spectrum of the same material at the nearest training speed. For held-out materials, use the nearest-neighbor material in photo-embedding space. Any learned method must beat this to justify itself.

Direct conditional regression. A network maps the photo plus speed and pressure to the log vibration spectrum. This is the strongest competitor for in-distribution speeds, because spectra are averaged quantities and do not suffer the blurry-mean problem the way height maps do. It gets the same image encoder, data, and hyperparameter search budget as our model.

Direct conditional diffusion on vibration spectrograms. This represents the "just generate both modalities" option. It is trained on the same data with matched compute.

Diffusion height plus a fixed physics filter. The HapticMatch height generator is paired with a fixed analytic contact filter and no learned $H_\theta$. This isolates whether learning the filter matters.

Ablations. Two ablations isolate our key ingredients:

  • Replace the diffusion height sampler with a U-Net trained with mean squared error, and keep the identical physics head. This tests whether preserving high-frequency height power matters for vibration.
  • Replace the explicit $f/v$ rescaling with speed fed as a learned embedding. This tests whether the speed equivariance is doing the work.

Reported numbers. For height quality, HapticMatch reports RF-DiT power-spectrum error as qualitatively lower than Pix2Pix, but no reported result for that frequency-domain MSE was found on Lacuna. We will rerun their released configuration as our height reference rather than quote a figure. There is no reported result on Lacuna for vibration prediction on this dataset.

What counts as winning. Beating the direct regressor on held-out speeds by a margin larger than seed variance, while staying within its error on interpolated speeds.

Why not just…?

Why not just regress the vibration spectrum from the photo and speed? This is the real threat. On speeds seen during training, it may match or beat our model, and we will report that if it happens. Its weakness is structural: nothing forces its output to shift in frequency with speed. The held-out-speed experiment settles the question. We train on the lower two-thirds of recorded speeds and test on the top third, then compare how quickly each model's error grows with extrapolation distance.

Why not just use a deterministic height regressor, since only the spectrum matters? Averaging over plausible textures removes high-spatial-frequency power He et al.. Under our mapping, that loss would show up as missing high-frequency vibration. The U-Net ablation measures this directly through band-wise vibration error. If the U-Net matches diffusion, the diffusion step is unnecessary and we will say so.

Why not jointly diffuse height and audio? With 2,000 image pairs and 27 recordings per material, a joint model has little data to learn speed behavior. It also gives no guarantee that the two outputs describe the same surface. The spectrogram-diffusion baseline and a cross-consistency metric show whether this concern is real in practice.

Proposed Experiments

HapticMatch splits. We test on three splits: held-out speeds, held-out pressures, and held-out materials.

Metrics:

  • Log-spectral distance and spectral-centroid error on vibration.
  • 2D power-spectrum MSE on height, using HapticMatch's metric.
  • A cross-consistency score: the error between recorded vibration and vibration predicted from the real GelSight map, compared with the error from the generated map. A small gap means generation preserves the physically relevant spectrum.

Controlled stage set. We evaluate time-domain rendering for variable-speed strokes using envelope correlation and spectrogram distance. The baseline is the direct regressor applied frame by frame using instantaneous speed.

Decisive checks:

  1. Whether the frequency-scaling exponent measured in the data is near 1 for a clear majority of materials.
  2. Whether our model's error at held-out speeds grows measurably more slowly than the direct regressor's.
  3. Whether swapping diffusion for regression in the height step raises high-band vibration error.

If the first check fails broadly, the premise is wrong and the project reduces to a negative result about geometry-driven vibration. That result is still worth publishing, given how haptic rendering pipelines currently treat geometry and vibration separately.