When two recordings of the same action are compared frame by frame, the grid of pairwise similarities shows the match as a bright, roughly diagonal stripe. A pause bends the stripe flat, a speed-up steepens it, and a reversed action turns it into an anti-diagonal. Two families of methods read this grid. Image-style networks look at the whole grid at once and are fast and robust to noise, but they place no constraint on which cells they treat as matched, so nothing stops them from rewarding a time-reversed match. Dynamic-programming methods trace a path cell by cell and guarantee that time moves forward in both sequences, but they are sequential, and their smoothed versions blur or collapse. We propose a model that keeps the image-style reading and changes only what it outputs. A 2D convolutional network reads the similarity grid. It then emits a short list of non-negative step sizes, and these are summed into a warping path. A path built this way is monotone by construction. Because the network is tied across transposition, the path is also symmetric between the two sequences. The model makes one forward pass and does no dynamic programming.
The bet is narrow and testable. On actions whose meaning depends on temporal direction, such as "pushing left-to-right" versus "right-to-left" in Something-Something V2, a convolutional matcher with a hard monotonicity guarantee should match or beat soft dynamic time warping in accuracy. It should beat unconstrained convolutional alignment on reversal and pause stress tests. And it should stay parallel. If a simpler hybrid matches it, we will say so.
Core Claim
Prior work leaves a specific gap. Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification (TAP) feeds the distance matrix and a time-offset matrix to a 3-layer CNN that predicts a free alignment-weight matrix. Its authors acknowledge that dropping monotonicity is a liability when order is absolute, and they make the score symmetric only by averaging the two directions. The few-shot video variant, Temporal Alignment Prediction for Few-Shot Video Classification, likewise predicts unconstrained weights $P$, and it trains in roughly half the time of a DTW-based module. ViSiL shows that a 4-layer CNN over a frame-similarity matrix recognizes warped, partial and slow-motion matches, but it outputs only a retrieval score with no path. On the other side, OTAM and D3TW use log-sum-exp relaxations of dynamic time warping (DTW) that are order-preserving. OTAM gets 42.8% 1-shot accuracy on SSv2 against 34.4% for CMN++, and it relaxes the boundaries so that buffer frames can be skipped. D3TW needs symbolic transcripts to avoid the soft path collapsing onto a single frame.
Our claim is that monotonicity and symmetry can be imposed on the output parameterization of a feedforward similarity-image reader rather than through a recurrence. We expect this to recover OTAM's order sensitivity and TAP's single-pass inference together, without transcripts and without averaging two directions after the fact.
Research Program
Given frame features for $X$ (length $N$) and $Y$ (length $M$), we form the cosine similarity grid $S$. A small 2D CNN, in the style of ViSiL, produces a feature map from $S$ and the time-offset channel borrowed from TAP. $K$ learned "step queries" cross-attend to this map and emit two logit vectors $u, v \in \mathbb{R}^K$. The path is
$$a_k = a_0 + (N - a_0 - a_{\text{end}}) \sum_{l \le k} \mathrm{softmax}(u)_l, \qquad b_k = b_0 + (M - b_0 - b_{\text{end}}) \sum_{l \le k} \mathrm{softmax}(v)_l.$$
Here $(a_k, b_k)$ is the $k$-th path point in frame coordinates. The offsets $a_0, a_{\text{end}}, b_0, b_{\text{end}} \ge 0$ are predicted start and end trims, which play the role of OTAM's relaxed boundaries. Because every increment is non-negative, both coordinates can only move forward. A pause in one sequence appears as near-zero increments on that axis.
For two or three rounds, each step query re-reads the CNN features sampled at its current path point. This lets the path bend far from the diagonal, where the fixed Gaussian prior of Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments breaks down. The refinement is a fixed, small number of parallel updates, not a recurrence over the grid.
The match score is $s(X,Y) = \frac{1}{K}\sum_k \tilde S(a_k, b_k) - \lambda \sum_k \max(0, \Delta_k - c)$. In this score, $\tilde S$ is the bilinearly interpolated similarity grid, $\Delta_k$ is the 2D length of step $k$, and the hinge with threshold $c$ and weight $\lambda$ charges a learned price for jumps. Jumping over cells is therefore a paid "gap", which is useful for skipping idle intervals. It is not a free way to dodge bad regions, which is how we address collapse without D3TW's transcripts.
Symmetry is exact. We run the head on $S$ and on $S^\top$, swap the two outputs of the second run, and average, so $s(X,Y) = s(Y,X)$ by construction. Training uses the standard episodic few-shot cross-entropy over scores, with gradients flowing into the frame encoder.
Baselines
The naive reference is mean-pooled frame features compared by cosine, the FewShot-TSN-style baseline that TAP's video paper reports losing to. If our model does not clearly beat it on SSv2, temporal structure is not being used at all.
The strongest order-preserving competitor is OTAM. Its reported SSv2 accuracy is 42.8% 1-shot and 52.3% 5-shot, and on Kinetics 73.0% and 85.8%. We rerun OTAM with our backbone, frame count, episode sampler and tuning budget rather than relying on those figures alone. Beating it means equal or higher SSv2 accuracy at lower wall-clock cost per episode. We time OTAM with a vectorized anti-diagonal implementation, since a naive loop would flatter us.
The strongest convolutional competitor is TAP. The file has no reported SSv2 number for either TAP paper (no reported result found on Lacuna for TAP on SSv2). It does report 83.5% 5-way 1-shot on UCF101 and 0.961 MAP on Spoken Arabic Digits, which we use as reproduction checks before rerunning TAP under our protocol. Beating TAP means higher SSv2 accuracy and a clear win on the reversal and pause tests below.
Three ablations isolate the key ingredient. The first keeps our exact CNN and attention head but has it output a free alignment matrix, as TAP does; this tests whether the monotone parameterization itself matters. The second drops transpose tying and instead averages the two directions after the fact; this tests whether exact symmetry matters. The third is the hybrid: our CNN refines $S$ into a cost grid, and soft-DTW reads the path. This last one is a serious competitor in its own right, discussed next.
Why not just…?
Why not just let the CNN clean up the similarity grid and run soft-DTW on top? This is the most serious threat, because it combines ViSiL-style noise tolerance with exact DP monotonicity. Our only structural advantages are parallelism and the absence of soft-min path smearing at large smoothing values. Few-shot SSv2 uses short clips, where DP cost is small. If the hybrid matches our accuracy there, the case rests on long sequences and on timing. We therefore also report accuracy-versus-time on Spoken Arabic Digits and on long synthetic warps. If the hybrid wins on both accuracy and acceptable time, we will report that the monotone head is not worth it.
Why not just add a monotonicity penalty to TAP's predicted matrix? A soft penalty trades off against the classification loss and can be overridden exactly where it matters, on hard reversed pairs. The free-matrix ablation with an added off-path penalty tests this directly. The test that decides it is the reversal check: under a hard constraint, a reversed query can never earn a high score. Under a penalty, it can.
Why not just use a fixed diagonal prior? The Gaussian-prior method reports 69.42% transductive on SSv2 and 93.33% on Kinetics. Its own authors note that the prior fails when warping pulls the path off the diagonal. The pause-injection test is designed to expose exactly that, and we compare in the inductive setting so that transductive refinement does not confound the result.
Why not just resample sequences linearly? Uniform stretching cannot represent a pause in one phase and a speed-up in another. LoCoMotif found that accounting for non-linear warping raised physiotherapy F1 from 0.75 to 0.88. We include linear resampling as a row in the warp-severity curve.
Proposed Experiments
The main benchmark is few-shot classification on SSv2, which is order-sensitive (The "Something Something" Video Database), with Kinetics as an order-insensitive control. On Kinetics we expect parity with OTAM, not gains, so any SSv2 gain should come from the order constraint rather than from extra capacity. Spoken Arabic Digits and MSR Action3D test non-video sequences under TAP's protocol.
The first decisive check is a reversal test. We take SSv2 direction-pair classes and reverse the query clips. A model that respects order should flip its prediction to the paired class. We measure the flip rate for our model, TAP, the free-matrix ablation and OTAM. We predict that ours and OTAM flip near-perfectly and that the unconstrained models do not.
The second check is pause and speed injection. We duplicate frames to simulate holds and drop frames to simulate bursts, at controlled severity. Accuracy is plotted against warp severity for all baselines, including linear resampling and the diagonal prior.
The third check measures path quality directly. Starting from UCR series and SSv2 clips, we apply known random monotone warps and score the predicted path against the true one by mean absolute frame error. We compare against hard DTW, soft-DTW at several smoothing values, and the hybrid. This also shows whether the gap hinge prevents collapse.
We report per-episode latency and memory against sequence length up to several hundred frames for all alignment methods on the same GPU.
The idea fails if any of three things holds. The free-matrix ablation might match us on the reversal and pause tests, meaning monotonicity does not matter. The hybrid might match us at comparable latency, meaning the parameterization adds nothing beyond speed. Or path error might exceed soft-DTW on known warps, meaning the head is not actually aligning.
References
- Temporal Alignment Prediction for Supervised Representation Learning and Few-Shot Sequence Classification
- Temporal Alignment Prediction for Few-Shot Video Classification
- ViSiL: Fine-grained Spatio-Temporal Video Similarity Learning
- Few-Shot Video Classification via Temporal Alignment
- D3TW: Discriminative Differentiable Dynamic Time Warping for Weakly Supervised Action Alignment and Segmentation
- Inductive and Transductive Few-Shot Video Classification via Appearance and Temporal Alignments
- LoCoMotif: discovering time-warped motifs in time series
- The "Something Something" Video Database for Learning and Evaluating Visual Common Sense