Replaying Motion with Sparse Displacement Handles

Video models that learn new action classes over time must keep a small buffer of old examples to avoid forgetting. The most memory-efficient recipe we know of stores one still frame per example and repeats it. Just a Glimpse: Rethinking Temporal Information for Video Continual Learning reports that this diversity-first buffer beats storing full clips by a wide margin. A still frame, however, cannot say which way things are moving. "Sitting down" and "standing up" share the same mid-action pose, as do "opening" and "closing," so a model rehearsed only on stills has no signal for keeping old direction-dependent classes apart.

We propose storing each still frame together with a few dozen displacement handles. A handle is a single point in the frame with the small motion vector observed there, costing a few hundred bytes per exemplar. At replay time, a fast local interpolation turns the handles back into a dense motion field. That field then drives a deformable convolution, a convolution whose sampling locations are shifted by learned per-pixel offsets, so the stored frame is warped into a pseudo next frame. The bet is that motion in action videos is concentrated on the actor and is spatially smooth, so a tiny, well-placed set of handles recovers almost all the direction information. If that holds, the buffer keeps the diversity of still-frame replay and also keeps the temporal cue.

Core Claim

At a fixed memory budget, a buffer of single frames plus optimally placed sparse displacement handles will preserve accuracy on direction-dependent old classes far better than single-frame replay. It will also stay within a small margin of single-frame replay on ordinary classes, because each exemplar grows by well under one percent of its storage. The handles must come from a selection procedure, not a uniform grid. With a few dozen points, a uniform grid mostly samples static background while the actor's hands and torso carry the motion.

The three research directions play distinct roles. Temporal cue deficiency is the problem being addressed. Deformable convolution is the representation of motion: a model's offset field between consecutive frames serves as its motion code, and the same offsets can warp stored features. Scalable global interpolation is a constraint and a method supplier. Handle selection and reconstruction run for every exemplar on every replay batch, so they have to be cheap. Selection reuses the Schur-complement reformulation from OptCtrlPoints, which decouples biharmonic weight computation from vertex count and cuts control-point search from about 1.6 days to 3 minutes. Reconstruction reuses partition-of-unity interpolation from Partition of unity interpolation using stable kernel-based techniques. That method solves small overlapping local kernel problems and blends them with weights summing to one, avoiding the cubic cost of a global kernel solve.

The closest prior work is Temporal Hallucinating for Action Recognition With Few Still Images. It infers missing motion for a still image by Gaussian-process interpolation over a video memory keyed by appearance. That approach cannot, even in principle, resolve the ambiguity we target. When two classes share the same static appearance, any function of appearance alone returns the same hallucinated motion for both. We therefore store a small amount of real motion instead of predicting it, and the interpolation runs over image space within one exemplar rather than over a global appearance space.

Research Program

The base model is a two-frame action recognizer. A 2D backbone produces features for frames $t$ and $t+\delta$. A lightweight offset head, of the kind that regresses deformable-convolution offsets, predicts a displacement field $D$ that aligns the two feature maps. The classifier sees appearance features, the field $D$, and the frame-$t$ features warped by $D$ through bilinear sampling. On live video at training and test time, $D$ comes from the offset head.

For a replayed exemplar, only frame $t$ is stored. The field is rebuilt from $K$ handles $\{(p_k, d_k)\}$, where $p_k$ is a space-time location and $d_k$ is the displacement recorded from the offset head when the exemplar entered the buffer. The reconstruction is

$$\hat D(x) = \sum_i w_i(x)\, s_i(x), \qquad \sum_i w_i(x) = 1,$$

where each $s_i$ is a kernel interpolant fit only to the handles inside overlapping patch $\Omega_i$, and the $w_i$ are compactly supported blending weights. Cost is linear in the number of feature-map query points and handles, rather than cubic in handles. This matters once handles extend over several time steps for longer actions.

Handles are selected greedily per clip to minimize the reconstruction error between $\hat D$ and the true $D$. The Schur-complement trick lets each candidate addition update the small $K$-sized system instead of refactoring a system over all feature locations.

Recording $d_k$ from the model's own offset head makes the handles a form of motion distillation. This has a side effect worth measuring. As later tasks change the offset head, stored handles act as fixed targets that anchor what old motion looked like. The cost per exemplar is $K$ handles, each holding three coordinates and two displacement components in half precision. For $K=32$ this is about 320 bytes against a compressed frame of tens of kilobytes.

The analysis has three parts. The first measures how direction information, defined as accuracy on time-reversed versus forward clips, scales with $K$ and with placement strategy. The second checks whether partition-of-unity reconstruction introduces patch-boundary artifacts that hurt gradients. The survey of partition-of-unity methods flags this risk for downstream learning, so we will compare it against a single global kernel solve at small $K$, where the global solve is cheap. The third tests whether learned offsets drift onto background, a known failure of deformable convolution, by checking how much handle mass lands inside annotated actor boxes.

Baselines

The naive reference point is single-frame replay as in Just a Glimpse, where the stored frame is repeated so that it carries no motion. The file reports that this method outperforms the prior best by 21.49% in average accuracy on the 20-task Kinetics benchmark of vCLIMB. It also reports backward forgetting of 2.59% on UCF101 against roughly 18% for baselines, while using 87.7% to 99.9% less memory. No reported absolute accuracy for this method was found on Lacuna, so we will reproduce it in our codebase. Beating it means a clear gain on direction-dependent old classes with no loss beyond noise on the remaining classes at the same byte budget.

The strongest competitor that also keeps motion is clip replay at equal bytes. Each exemplar stores two or more frames, so fewer exemplars fit, which is the configuration the Glimpse paper argues against. We also include a frame plus a uniform low-resolution displacement grid, upsampled bilinearly, with the same byte count as $K$ handles. This is the most dangerous baseline because it uses identical information types and differs only in placement and interpolation.

An appearance-only hallucination baseline follows Temporal Hallucinating for Action Recognition With Few Still Images. It predicts the motion field from the stored frame's features using a memory of real fields from the current task. This baseline tests whether stored motion is needed at all.

The key ablations hold $K$ fixed and change only placement: optimized, farthest-point, random, or uniform grid. They then hold placement fixed and change only reconstruction: partition of unity, global kernel, or nearest-handle. OptCtrlPoints reports that optimized placement cuts fitting error by more than 50% versus farthest-point sampling in 3D deformation. We expect a similar ordering here, but it has to be shown on motion fields. All methods share the backbone, task order, total buffer bytes, number of replay steps, and hyperparameter search budget, with three seeds each.

Why not just…?

Why not just store two frames per exemplar? This is the direct alternative, and our equal-byte clip-replay baseline is exactly this test. Doubling frames halves the number of exemplars at a fixed budget. The Glimpse result indicates that exemplar diversity matters more than temporal depth for overall accuracy. Handles aim to recover the temporal part for under one percent extra storage. If two-frame replay matches us on direction-dependent classes and loses little overall, our method offers no advantage. The budget sweep will show where, if anywhere, that crossover happens.

Why not just store a coarse optical-flow grid? This is the real threat. A 4×4 grid holds about as many numbers as 16 handles and needs no selection step. Our argument is that action motion is localized, so a coarse uniform grid averages the moving hand with the static background around it. That is an empirical claim, settled by the matched-byte grid baseline and by the handle-placement ablation. If the grid matches optimized handles, we will report the simpler method as the recommendation and treat the selection machinery as unnecessary.

Why not just store one global motion embedding vector per exemplar? A pooled vector, such as the mean feature difference between frames, is even cheaper and may encode direction well enough for classification. However, it cannot be used to warp the stored frame, so the motion pathway of the model never sees spatially structured replay input. We include it as a baseline and expect it to help direction classes less than spatial handles. If it does not, the deformable-warping component of our method is unnecessary.

Why not just hallucinate motion from the frame, as earlier still-image work did? This works for recognizing still images in general, where HVM reports 35.4% versus 26.6% on WEB101 with one image per class. Direction pairs are the case where appearance carries no answer by construction, so we predict this baseline will match single-frame replay on those pairs. The experiment will confirm or refute that prediction.

Proposed Experiments

The main benchmark is class-incremental vCLIMB on Kinetics and UCF101, the settings used by the Glimpse paper. We add a direction-focused split from the RGB videos of NTU-RGB+D. Classes are chosen as pairs that differ mainly in temporal direction, such as sitting down and standing up, and paired classes are placed in different tasks so the old member must be remembered through replay. A model-agnostic diagnostic accompanies every run. Each test clip is classified forward and time-reversed. A model that retains temporal cues should change its prediction for direction pairs and keep it for direction-symmetric classes. The fraction of old-class pairs where this happens is our main temporal-retention metric.

We will report average accuracy, backward forgetting, accuracy on direction-pair classes, buffer bytes, and replay-time overhead per batch. We will also sweep $K$ from 8 to 128 and buffer size across a decade.

The decisive check is the matched-byte comparison between optimized handles and the uniform displacement grid on direction-pair classes. A gap of several points at $K \le 32$ that closes as $K$ grows would confirm that placement is what matters. No gap would refute the selection component. A second decisive check is the forward/reversed diagnostic for single-frame replay against handle replay. If handle replay keeps the reversal sensitivity on old tasks while single-frame replay loses it, sparse motion storage resolves the conflict between diversity-first replay and temporal cue deficiency.