Heat-Priced Predictive Frame Skipping for Tactile Slip Perception on Edge Chips

A robot that holds an object by touch has to watch for slip continuously, because slip can start and finish within tens of milliseconds. With camera-based touch sensors, the expensive part is not the small recurrent network that tracks slip over time. It is the per-frame image processing that turns each tactile image into features, run at 30 to 140 frames per second on every finger pad. On an embedded chip this steady load produces heat. On modern chips heat does more than waste energy: static leakage current, which flows even when transistors are not switching, rises exponentially with temperature. Warming therefore raises power, which raises temperature further, until the chip cuts its own clock speed. When that happens, inference latency jumps unpredictably, and a slip controller running at 30 Hz misses the window it needs to save a grasp.

We propose to let the recurrent slip model decide when the next tactile frame is worth processing. It makes that decision from two things: its own prediction of how likely slip is in the near future, which depends on the robot's planned motion, and the price of computing right now, which rises with die temperature in the same exponential way leakage does. During a stable hold the model coasts on its internal state and the chip idles and cools. Before a planned acceleration, or as soon as the state shows early signs of slip, it wakes the encoder at full rate. The core claim can be tested directly. Gating that is predictive (driven by planned actions and the recurrent state) and priced by temperature should keep slip-onset detection latency and slip-velocity accuracy close to an always-on pipeline, and should avoid the throttling events that always-on processing triggers at high ambient temperature. It should do this better than reactive gating driven by frame differences, and better than a fixed, conservative frequency cap.

What exists and what is missing

Recurrent tactile slip models already work well offline. Proactive slip control feeds tactile shear forces from a 4x4 magnetic array, together with the robot's planned future actions, into an LSTM, a gated recurrent network that keeps a running memory. The network predicts upcoming slip, and a 30 Hz controller reshapes the trajectory; the authors report preventing drops in regimes where the unassisted robot failed 95% of the time. Learned Slip-Detection-Severity extracts physically meaningful features from GelSight Mini marker displacement fields: net velocity, divergence and curl. It uses an LSTM to regress slip velocity with a mean absolute error of 0.26 cm/s. Both papers run on unconstrained host machines and report nothing about latency, energy or temperature. These two papers motivate the proposal and supply our perception models. Nazari et al. in particular supply the idea we reuse: conditioning on planned actions lets a model anticipate slip instead of reacting to it.

Systems work has studied the heat–leakage loop, but only for batched feedforward models. Thermal-aware on-device inference models power as dynamic plus temperature-dependent leakage on a Jetson Nano. Its method (TADF) picks the highest sustainable clock that stays under 85°C at 40°C ambient, and it splits CNN layers across CPU and GPU, cutting AlexNet and VGG16 latency by 21–43%. PolyThrottle tunes GPU, CPU and memory frequencies with constrained Bayesian optimization, saving up to 36% energy per query. It also documents a "minimum frequency paradox": clocking too low costs more energy, because the chip never gets to finish quickly and sleep. Accurate and stable run-time power modeling gives a temperature-aware leakage model with 2.8–3.8% error on ARM cores. Leakage-aware scheduling shows that alternating fast and slow execution lowers peak temperature, and that switching too often erases the gain. These papers supply our power model and our strongest systems baselines, and they constrain the design in two ways. The paradox rules out "just run slowly." The switching-overhead result says gating decisions should come in bursts, not flip every frame.

The recurrent-network line supplies the adaptive-computation idea. Time Adaptive Recurrent Neural Network (TARNN) learns an input-dependent time constant, so the network can effectively ignore uninformative steps. It solves a synthetic long-dependency task at 100% accuracy where an LSTM reaches 45%. The limitation matters for our setting. TARNN skips state updates, but it still reads and encodes every input. On tactile streams the encoder is the dominant cost, so input-level pondering by itself saves little heat. Our gate moves the skip decision upstream, to whether the encoder runs at all.

Core mechanism

At each sensor tick $t$ the model holds a recurrent state $h_t$, the current die temperature $T_t$ and the planned end-effector actions over a short horizon. A small gate head outputs a probability $g_t$ of running the tactile encoder on the incoming frame. If the gate is closed, the recurrent cell advances using a learned "no observation" input, as in continuous-time networks that handle irregular sampling, so the model still produces a slip estimate every tick. We train with

$$\mathcal{L} = \mathcal{L}_{\text{slip}} + \sum_t \mu_0\, e^{\beta (T_t - T_0)}\, g_t\, c_{\text{enc}},$$

where $\mathcal{L}_{\text{slip}}$ is the slip detection and velocity loss, $c_{\text{enc}}$ is the measured energy of one encoder call, $T_0$ is a reference temperature, and $\beta$ is fitted from the chip's leakage-versus-temperature curve using the Walker-style model. The exponential form comes directly from subthreshold leakage scaling. During training, $T_t$ and $\mu_0$ are randomized so that a single network learns to trade accuracy for compute across temperatures. At deployment the real sensor reading drives the price. A minimum-dwell constraint (the gate stays in a state for at least $k$ ticks) respects the switching overheads Huang et al. identify. It also produces burst-then-idle patterns that let the chip enter sleep states instead of running at a low, leaky clock.

Research program

The first step is a characterization study that could falsify the premise. We will stream tactile data at recorded rates through the Jawale-style pipeline, which combines marker tracking, field features and an LSTM, on a Jetson-class board in a controlled enclosure at 25°C and 40°C ambient. We will measure die temperature, the split between leakage and dynamic power, throttling events and per-frame latency. If one GelSight pad at 30 Hz never approaches throttling, we will scale to the regimes where the problem plausibly bites: 140 fps as in PyTouch, and many-pad streams replayed from DECO-50, which has 17 pads per hand on two hands. We will report honestly where the thermal problem begins. The second step trains the gated model on offline slip data. The third step closes the loop on a real arm with the edge board running perception, repeating Nazari-style held-object motions under high ambient temperature.

Baselines

The naive reference is the always-on pipeline: every frame encoded, LSTM updated every tick, default operating-system clock governor and hardware throttling. This is how the published slip models would behave if moved onto the robot unchanged. It sets the accuracy ceiling, and it shows whether throttling-induced deadline misses happen at all.

The strongest systems competitor is always-on processing under a proactive frequency cap, following the TADF algorithm of Zhang et al.. We add a PolyThrottle-style Bayesian search over CPU, GPU and memory clocks, tuned for a 30 Hz deadline. Their reported result is a 21–43% CNN latency reduction while staying under 85°C at 40°C ambient; no reported result was found on Lacuna for recurrent tactile workloads under these methods. This baseline gets the same search budget as our gate's hyperparameter search: the same number of on-board trials, using PolyThrottle's reported 5 to 15 samples as the floor. To beat it, we must match its zero-throttling behavior while achieving lower mean energy and equal or better slip-onset latency, or we must sustain a higher sensor rate or pad count before any deadline miss.

The strongest perception competitor is a heuristic event gate. It runs the encoder only when the mean absolute frame difference exceeds a threshold, and it may make that threshold temperature-dependent. The tactile work on frame differencing, AnyTouch 2, suggests this signal is informative. The threshold will be tuned on the same validation data to the same energy budget as our gate. A TARNN-style model that learns time constants but encodes every frame isolates whether skipping the encoder, rather than skipping state updates, is what saves heat. Two ablations of our gate isolate its key ingredients: one removes the planned-action input, making the gate purely reactive, and one uses a fixed price in place of the temperature-dependent one. Beating the event gate means lower slip-onset detection latency at matched energy on trials where slip is caused by planned accelerations.

Why not just…?

Why not just gate on frame differences? This is the real threat. On slow, spontaneous slip it may match a learned gate. Its weakness is structural: it wakes only after the image has changed, so on accelerations it pays one reaction delay at exactly the moment Nazari et al. show anticipation matters most. The matched-energy comparison of slip-onset latency, split by trials with and without planned acceleration, settles this. If the heuristic ties us on both kinds of trial, we will report that the action-conditioned gate is unnecessary, and the contribution narrows to the temperature-priced budget and the characterization.

Why not just run a smaller model always on? Shrinking the recurrent network saves almost nothing, because the LSTM on a handful of field features is already tiny. The cost lies in marker tracking and image encoding, which the characterization will quantify directly. Shrinking the encoder is compatible with our approach and will be applied to every method alike. The published accuracy (0.26 cm/s MAE, 92% detection) sets how far the encoder can be degraded before the perception task suffers.

Why not just cap the clock at a sustainable frequency? A cap prevents throttling, but it spends the same compute on a motionless hold as on a slip transient. PolyThrottle's minimum-frequency paradox suggests that a lower always-on clock leaks more energy than bursting and sleeping. The capped baseline measures this directly. If it matches our gate on both energy and headroom, it wins and we will say so.

Why not a transformer over a fixed window of frames? A windowed model must re-encode or buffer many frames at every step. DECO reaches 82.50% success on DECO-50, but with a 91M-parameter model. That model is a reasonable offline reference, but it cannot be a streaming baseline on a Nano-class board, and we will report whether it can hold 30 Hz at all.

Decisive checks

The evaluation uses three kinds of data. For slip velocity and detection we use the GelSight Mini deformation-field protocol, recollected with the same objects and procedure if the original data are not released. For proactive slip under planned motion we use the PSC-style protocol with held cans and liquid-filled containers. For multi-pad scaling we replay DECO-50. The metrics are slip-velocity MAE, slip-onset detection latency, fraction of missed 30 Hz deadlines, peak die temperature, time spent throttled, energy per second and, in closed loop, drop rate and object rotation time steps.

The proposal succeeds if three things hold. At 40°C ambient the always-on pipeline shows measurable throttling or deadline misses at some realistic rate or pad count. At that operating point our gate stays within a small margin of always-on MAE with zero deadline misses. And it beats the frame-difference gate on onset latency during planned accelerations at matched energy. It fails, and we would report the failure, if heat never becomes binding at realistic loads, or if the tuned heuristic gate matches it.