Few-shot action recognition asks a model to recognize a new human action from one to five labeled video clips. Sometimes the order of frames is the whole story. "Opening a drawer" and "closing a drawer" look nearly identical frame by frame and differ only in direction. Other times order barely matters, because "playing guitar" is recognizable from a single frame. Current methods fix this choice ahead of time. Some always enforce strict chronological alignment. Some ignore order entirely. One balances the two with a single hand-tuned number per dataset. We propose instead that the comparison function between a query video and a class should read the support set (the handful of labeled examples in the episode) and predict how strictly to enforce temporal order for each class. Strict order would apply when the classes in the episode can only be told apart by direction, and loose order when appearance already separates them.
Concretely, we build one time-aligned prototype sequence per class by warping the support videos onto a shared timeline. We then score queries against it with an optimal-transport alignment, a soft matching of query segments to prototype segments, whose penalty for out-of-order matches comes from a small learned network. That network sees only the support set, so every query in an episode is judged by the same per-class metric. This keeps the cost and consistency advantages of ordinary prototypes while making the metric itself adaptive. The test is simple to state. On episodes that mix order-critical and order-agnostic classes, an adaptive strictness should beat both the best fixed setting and the best order-free method.
Core Claim
The disagreement in the literature about the "arrow of time" is a disagreement about a per-class quantity that the support set can estimate. OTAM aligns query and support frames with a differentiable dynamic time warping that enforces monotonic order. TRX pools ordered frame tuples without any global alignment path and beats OTAM on Something-Something V2. CPM discards order for its focused prototypes, and its authors concede this can confuse "putting A on B" with "taking A off B." CMOT is the clearest evidence that the answer varies. It writes the video distance as optimal transport over a cost $C = C^{SE} + \alpha C^{PO}$, where $C^{SE}$ is semantic distance between segments, $C^{PO}$ penalizes matching segments at different time positions, and $\alpha$ is a manually tuned scalar. It finds that appearance-heavy UCF101 wants small $\alpha$ and temporally heavy SSv2 wants large $\alpha$.
We claim $\alpha$ should be a function of the episode and the class, $\alpha_k = g_\theta(\mathcal{S})$, rather than a dataset constant. The order-penalty strength $\alpha_k$ for class $k$ is predicted by a network $g_\theta$ from the whole support set $\mathcal{S}$. Making $\alpha_k$ episode-conditioned rather than query-conditioned is deliberate. Query-adaptive prototypes such as TRX's couple the class representation to a single test sample. That breaks consistency across queries and multiplies cost by the number of queries. A support-conditioned metric avoids both problems.
Research Program
The model has three parts on top of a frame-level backbone. To match OTAM and TRX, this is an ImageNet-pretrained ResNet-50 over 8 sampled frames.
The first part is a temporal class prototype. For a class with $K$ support videos, we soft-align every support video to a medoid support video using OTAM's log-sum-exp dynamic time warping. We then average the aligned segment features, which gives a prototype sequence rather than a single averaged vector. This adapts the Prototypical Networks idea of a class mean to sequences without collapsing time. It also addresses a limitation shared by OTAM, CMOT and TARN: all three compare the query to each support video separately and only average scores in the multi-shot case. In 1-shot, the prototype is simply the support video.
The second part is the strictness predictor $g_\theta$, the learned element of the metric. Its inputs are cheap statistics computed inside the support set only. The main one is a reversal sensitivity for each pair of classes: how much more similar class $k$'s prototype is to class $j$'s prototype played backwards than forwards, measured with order-free matching. A high value means the two classes are near time reversals of each other, so order must decide between them. Other inputs are the spread between ordered and unordered alignment cost across classes, and, for $K>1$, how consistently the support videos within a class align monotonically to one another. $g_\theta$ is a two-layer network over these statistics. It outputs one $\alpha_k \ge 0$ per class and is trained end-to-end with the episodic cross-entropy loss.
The third part is the scoring rule. The query's distance to class $k$ is the entropic optimal-transport cost, solved with Sinkhorn iterations as in CMOT, under the cost matrix $C_k = C^{SE} + \alpha_k C^{PO}$. Following Meta-Baseline, which found that changing the metric between pre-training and episodic use hurts transfer, the semantic term uses the same scaled-cosine distance as the pre-training stage. The training recipe is therefore whole-classification pre-training followed by episodic fine-tuning.
The analysis program asks three questions. Does the predicted $\alpha_k$ track a measurable property of classes, namely the drop in accuracy when frames are reversed? TRX reports a drop from 59.1% to 51.3% on SSv2 under reversal, which shows this property is real at the dataset level. Does the predicted strictness transfer across datasets without retuning? And where does the gain come from: the prototype sequence, the adaptive metric, or their combination?
Baselines
The naive reference point is a Prototypical Network on temporally averaged frame features with squared Euclidean distance, as in Prototypical Networks. It is run with our backbone, frame sampling and pre-training. It indicates how much any temporal modeling buys on each benchmark. VIM reports that its alignment method beats this kind of ProtoNet by nearly 10% in 5-shot accuracy on ActivityNet.
The strongest competitor is TRX. It builds query-specific prototypes from frame tuples, does not enforce global order, and can therefore absorb order-sensitivity implicitly through its tuples. It is the method most likely to make an explicit strictness predictor unnecessary. TRX reports 64.6% accuracy on SSv2 and 85.9% on Kinetics, a 12% improvement over OTAM on SSv2. The file does not state the shot setting for these numbers, so we will rerun TRX from its released configuration in both 1-shot and 5-shot. Beating TRX means higher 5-way accuracy than our reproduction under identical backbone, frame count, pre-training, episode sampler (10,000 test episodes with 95% confidence intervals) and an equal hyperparameter search budget. The bar is a statistically clear win on SSv2 and on mixed-domain episodes, while staying within its confidence interval on Kinetics.
The most direct competitor is CMOT with $\alpha$ tuned per dataset on validation classes. It is our model without the predictor. CMOT reports 46.8% 1-shot on SSv2, 90.4%/95.7% 1/5-shot on UCF101 and 66.9%/81.5% on HMDB51. OTAM is the strict-order baseline: 42.8%/52.3% on SSv2 and 73.0%/85.8% on Kinetics. CPM reports 57.1% 1-shot on SSv2 and 81.0% 1-shot on Kinetics. CPM relies on Mask-RCNN object boxes, so we report its published numbers but compare head-to-head only in a variant without objects, and we state this asymmetry openly.
Ablations isolate the key ingredient. They are: a single globally learned $\alpha$; $\alpha$ fixed per dataset; our per-class episode-conditioned $\alpha_k$; a query-conditioned $\alpha_{q,k}$, which tests whether coupling to the query buys anything at its extra cost; and an oracle $\alpha$ chosen per test class on test labels, which bounds the possible gain from adaptivity. A further ablation replaces the prototype sequence with per-support scoring averaged afterwards, which separates the prototype contribution from the metric contribution.
Why not just…?
Why not just tune $\alpha$ per dataset, as CMOT does? This is the real threat. If benchmarks are homogeneous in how much order matters, a dataset constant captures nearly everything. CMOT's own results show the right $\alpha$ differs between UCF101 and SSv2, which suggests it also differs between classes inside a dataset, but nothing in the file measures that within-dataset variation. Two experiments settle it: the gap between per-dataset $\alpha$ and the per-class oracle $\alpha$, and performance on mixed-domain episodes. If the oracle gap within SSv2 is under about one point, we will report that per-class adaptivity is unnecessary and reduce the claim to cross-domain robustness.
Why not just use TRX, whose tuple attention already learns order cues? TRX's 59.1%-to-51.3% reversal drop shows it uses order. Its tuples are local, though, and it has no global alignment, so it cannot represent "this whole class must unfold in this direction." Its prototypes are also rebuilt for every query. We test the first point directly on episodes made of time-reversal class pairs, such as opening and closing the same object. The prediction is that our model's margin over TRX is largest there and disappears on appearance-dominated classes. If TRX matches us on reversal-pair episodes, the proposal's central claim is falsified.
Why not just learn a richer fixed metric, such as a bilinear similarity or a query-generated network? COLA shows that a learned bilinear similarity beats cosine for audio representations, and Hypencoder generates a query-specific scoring network for retrieval. A positive semidefinite bilinear form, however, amounts to a linear projection followed by Euclidean distance, which TRX's value projection already provides. A per-query generated network also brings the overfitting and latency costs that the research file lists as open problems. We include a bilinear semantic cost as an ablation. We expect it to change little, because the missing piece is a decision about temporal structure, not a better frame-to-frame distance.
Why not just ensemble an ordered score and an unordered score with a fixed weight? This is a cheaper form of the same dataset-constant assumption. It is covered by the per-dataset $\alpha$ ablation and the mixed-episode test.
Proposed Experiments
We train on the standard few-shot splits of SSv2, Kinetics-100 (64/12/24 classes from CMN), UCF101 and HMDB51 and evaluate 5-way 1-shot and 5-shot accuracy. The decisive checks go beyond the standard tables.
First, we build mixed-domain episodes by training on the union of SSv2 and Kinetics base classes and testing on episodes that draw classes from both novel splits. No single dataset-level $\alpha$ fits these episodes, so this is where an episode-conditioned metric must earn its keep.
Second, we build reversal-pair SSv2 episodes from novel classes that are approximate time reversals of each other, alongside control episodes of order-agnostic classes. We measure our model's margin over TRX and over per-dataset CMOT in each.
Third, we test whether the predicted $\alpha_k$ is meaningful. For each novel class we correlate it, using rank correlation, with that class's accuracy drop under frame reversal, computed independently with a frozen order-free model.
Fourth, we report wall-clock cost per episode as a function of queries per class. Because the metric is fixed per class within an episode, our cost should grow more slowly than TRX's query-specific prototype construction.
The proposal succeeds if adaptive strictness beats both the per-dataset setting and TRX on the first two checks, recovers a substantial share of the oracle gap, and shows a positive correlation on the third. It fails informatively if the oracle gap is negligible, which would mean temporal strictness really is a property of datasets rather than of classes.
References
- Few-Shot Video Classification via Temporal Alignment
- Temporal-Relational CrossTransformers for Few-Shot Action Recognition
- Compound Prototype Matching for Few-Shot Action Recognition
- Few-Shot Action Recognition with Compromised Metric via Optimal Transport
- Prototypical Networks for Few-shot Learning
- TARN: Temporal Attentive Relation Network for Few-Shot and Zero-Shot Action Recognition
- Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning
- Contrastive Learning of General-Purpose Audio Representations
- Hypencoder: Hypernetworks for Information Retrieval