A single photo of someone halfway into a chair looks the same whether they are sitting down or standing up. Vision models trained on still frames therefore confuse actions that are time-reversals of each other, such as opening and closing, picking up and putting down, or falling and sitting. When labels are scarce, the problem gets worse, because a few labeled snapshots cannot teach the model which way motion is going. Unlabeled video holds exactly this information, and the field already has a good tool for turning it into a training signal: proximity-weighted contrastive learning. This trains an encoder so that its embeddings are close when the inputs are physically close, and the closeness is graded rather than all-or-nothing. Applied to time, though, every such loss we know of is symmetric. A frame half a second before the anchor counts as exactly as close as a frame half a second after it. Such a loss encodes the time-reversal symmetry that causes the ambiguity in the first place.
We propose to break that symmetry on purpose. A still-frame encoder is pretrained on unlabeled video so that, from one frame alone, a small predictor must pick out the frame $\delta$ seconds in the future. The frame $\delta$ seconds in the past, equidistant in time and nearly identical in appearance, serves as the hardest negative. The encoder can only satisfy this objective by finding cues in a single image that do not survive reversal: lean and balance, motion blur, gravity acting on objects, the state of a hand relative to its target. We then test whether this direction-aware embedding makes few-shot action recognition from still images better specifically on time-reversed action pairs. These are the cases where static models are known to fail and where the closest prior methods give no guarantee.
Core Claim
Distance-preserving contrastive objectives are the right tool for label-free structure. When the distance is temporal, though, preserving distance (an unsigned quantity) throws away direction (the sign). Replacing the past frame's role as a near-positive with a role as a hard negative for forward prediction should yield a still-frame representation that carries an estimate of motion direction. That information, in turn, should close much of the gap between single-frame and multi-frame models on reversal pairs under 1 to 10 labels per class. It should do so without optical flow, without dense clips at test time, and without an external video memory.
This positions the four directions unevenly. Temporal cue deficiency is the target problem. Label-efficient learning is the regime. Spatial proximity embedding and spatial contrastive learning supply the mechanism, which we modify. The modification is the contribution, and the symmetric version is the central ablation.
Relation to Prior Work
Three methods supply the continuous-proximity machinery we reuse. WorldComp2D weights a contrastive loss by the physical distance between local image views. TRIDENT regresses embedding differences to true spatio-temporal path distances across several kernel widths to avoid dimension collapse. HierLoc scales contrastive penalties by great-circle distance. All three preserve a symmetric metric, and none operates on still frames that must stand in for missing motion. We borrow the graded weighting and the multi-kernel idea and add a signed term they lack. Two other papers constrain our sampling design. Tile2Vec shows that a hard neighborhood radius is brittle, and OmniSat shows that nearby samples should not be pushed apart as random negatives. For both reasons, we downweight same-video near-neighbors continuously rather than excluding them with a binary filter.
Two methods motivate the problem and form the strongest prior attacks on it. Chen and Grauman retrieve similar poses in unlabeled video and synthesize preceding and succeeding poses for a few labeled snapshots. HVM hallucinates motion features for 1- to 10-shot still-image recognition with a Gaussian process over an external video memory. Both rely on appearance similarity to find matching video. Because they retrieve by appearance, the sit-down and stand-up instances of the same posture match the same video, and the hallucinated context does not resolve direction. Our encoder must internalize direction at training time and needs no lookup at test time.
Three results sharpen what has to be shown. SMILE finds that single stored frames beat multi-frame clips for memory-limited continual video learning, but concedes failure on direction-dependent classes. Something-Something shows 2D models at 76.5% error on a 10-class motion subset versus 44.9% with temporal modeling. Something-Else shows that tracked box geometry improves 5- and 10-shot recognition by over 12% by suppressing appearance bias, at the price of dense supervised boxes. Together these define the target: SMILE's single-frame efficiency, without its blindness to direction.
Research Program
Pretraining objective. An image encoder $f$ maps a frame $x_t$ to an embedding $z_t$. A small forward predictor $h$ maps $z_t$ to a guess of the embedding $\delta$ seconds ahead. With cosine similarity $s$ and temperature $\kappa$, the forward term for one anchor is
$$ \mathcal{L}_{\text{fwd}} = -\log \frac{e^{s(h(z_t), z_{t+\delta})/\kappa}}{e^{s(h(z_t), z_{t+\delta})/\kappa} + e^{s(h(z_t), z_{t-\delta})/\kappa} + \sum_{n} w_n\, e^{s(h(z_t), z_n)/\kappa}}, $$
The sum runs over other frames $z_n$ in the batch. The weight $w_n = 1 - e^{-|\Delta t_n|/\tau}$ shrinks toward zero for frames from the same video close in time, so they are not treated as full negatives, and equals one for frames from other videos. The mirrored past frame $z_{t-\delta}$ always enters at full weight.
Alongside it we keep a symmetric proximity term: the proximity-weighted contrastive loss of WorldComp2D applied over $|\Delta t|$ with several bandwidths $\tau$, as in TRIDENT. This term learns phase within an action, and the forward term learns direction. The total loss is $\mathcal{L} = \mathcal{L}_{\text{sym}} + \lambda \mathcal{L}_{\text{fwd}}$, and $\delta$ is sampled from a range (roughly 0.1 to 1 second) so that the encoder cannot specialize to one speed.
Data and shortcut control. We pretrain on unlabeled frames from Kinetics and Something-Something V2 videos, with labels discarded. The objective invites shortcuts: fades, camera zoom drift, and compression patterns that correlate with position in a clip. We therefore sample anchors away from clip boundaries, apply identical photometric augmentation to the triplet, and run a leakage probe that tries to predict absolute frame index from $z_t$. If that probe succeeds well beyond what the symmetric model achieves, the direction gains are suspect.
Downstream use. At test time only $f$ is used, on a single image. Few-shot classifiers (linear probe and prototype classifier) are trained on 1, 5, or 10 labeled still images per class.
Analysis. Beyond accuracy, we ask how much direction information a still frame contains at all. A linear probe predicts the sign of motion between held-out frame pairs from one frame's embedding, with results reported per action family. If even this objective cannot push the probe above chance for some families, that bounds what any still-image method can do there. That bound is a useful negative result on its own.
Baselines
The naive reference is an ImageNet-pretrained image encoder of the same architecture, used as a frozen feature extractor with a linear probe on the few labeled images. It shows how far appearance alone goes. In Tile2Vec, such features (47.7%) trailed a domain-specific self-supervised model (52.6%) on land cover. We expect them to sit near chance on reversal pairs.
The strongest existing method for the label-scarce still-image setting is HVM. It reports 1-image accuracy of 35.4% on WEB101 (versus a 26.6% baseline), 42.2% on VOC 2012 Actions (versus 39.0%), and 60.2% on DIFF20 unseen classes. We compare against these numbers on the same splits. We also reimplement HVM's memory with the same unlabeled video pool and backbone family we pretrain on, so that gains cannot come from more or different video. Beating HVM means higher 1- and 5-shot accuracy on WEB101 and VOC with no test-time memory. More importantly, it means a clear margin on the reversal-pair diagnostic below, for which no reported result found on Lacuna for HVM or any other method.
The key ablation is the identical model trained with only the symmetric proximity term ($\lambda = 0$), with the same data, steps, augmentation and tuning grid. This isolates the signed term. Our hypothesis predicts the two models match on ordinary still-image action classes and diverge sharply on reversal pairs. If they also match on reversal pairs, the core claim is false.
A distillation baseline trains the same image encoder to regress the features that a frozen self-supervised video model computes on the short clip centered at each frame. This is the most direct alternative way to push motion knowledge into a still-frame network. It receives the same pretraining video and compute.
An arrow-of-time pretext baseline trains the image encoder, through a two-frame head, to classify whether an ordered pair of frames runs forward or backward, and then discards the head. It gives the encoder the same sign information as our method through a classification loss rather than a contrastive one.
All baselines share the backbone, pretraining budget (fixed GPU-hours), few-shot protocol, and an equal hyperparameter search budget. Results are averaged over at least five sampled few-shot episodes sets with confidence intervals.
Why not just…?
Why not just use a video-pretrained backbone and feed it the still image repeated as a clip? A repeated frame is SMILE's "boring video," and SMILE itself reports that such inputs fail on direction-dependent classes. A video model's temporal filters see zero motion, so the input carries no direction signal. Something-Else additionally shows heavy appearance backbones like I3D lose about 15% when appearance shortcuts are removed. We include this as an extra reference row on the reversal-pair diagnostic. We expect it at chance there.
Why not just distill a video model's clip features into an image encoder? This is the real threat, because it transfers motion knowledge without any new objective. The open question is whether a self-supervised video teacher's features distinguish forward from reverse at all. Reconstruction-style video pretraining has no reason to make forward and reversed clips map apart, and a symmetric teacher yields a symmetric student. The distillation baseline above settles this directly. We also run a sanity check: if the teacher's own embeddings of a clip and its reversal are nearly identical, distillation cannot help on reversal pairs. If distillation matches our method there, we will report that the simpler route suffices.
Why not just train an arrow-of-time classifier and use its features? It supplies the same sign label. Binary classification on frame pairs, however, can be solved by comparing the two inputs. The network can detect which frame has the more open hand without either frame alone encoding direction. Our forward predictor sees only $z_t$, which forces single-frame encoding. The arrow-of-time baseline, together with the single-frame direction probe, tells us whether this distinction matters in practice. If the arrow-of-time encoder's single-frame probe is as good as ours, the contrastive structure is unnecessary.
Why not just add optical flow or frame differences? For the target setting there is no second frame. HVM notes that flow is undefined for single photographs. Even with video, ACT-detector reaches 25 to 30 fps only when flow computation is excluded. Flow-based models serve here as an upper reference on the video version of the diagnostic, not as a competitor.
Proposed Experiments
The first experiment is standard few-shot still-image action recognition on WEB101, VOC 2012 Actions and DIFF20 at 1, 5 and 10 images per class, following HVM's protocol and comparing to its reported numbers.
The second, decisive experiment is a reversal-pair diagnostic built from Something-Something V2. We select class pairs that are time-reversals (opening versus closing, folding versus unfolding, picking up versus putting down, and similar), take single frames from the middle third of each clip where state cues are weakest, and run 1-, 5- and 10-shot binary classification. Chance is 50%. As an upper reference, the same pairs are evaluated with a multi-frame model, mirroring the 76.5% versus 44.9% single- versus multi-frame gap that Something-Something reports on its 10-class subset. The prediction is specific: ours clearly above both the symmetric ablation and the ImageNet probe on this diagnostic, while all three stay close on the non-reversal classes of experiment one. We repeat the same diagnostic on sit-down versus stand-up style frames from J-HMDB. To rule out reliance on start or end states, we report accuracy as a function of frame position within the clip.
The third experiment returns to SMILE's setting. In single-frame replay continual learning on UCF101, we swap in our encoder and measure accuracy on the direction-dependent classes SMILE flags as failures. This tests whether direction-aware still frames recover what single-frame replay gives up, while keeping its memory savings.
The project fails cleanly if the symmetric ablation matches the full model on reversal pairs, if distillation or the arrow-of-time pretext matches it at equal compute, or if the frame-index leakage probe explains the gains. Any of these outcomes is informative. The last also gives a measured answer to how much "which way" a single image can hold.
References
- WorldComp2D: Spatio-semantic Representations of Object Identity and Location from Local Views
- TRIDENT: Cross-Domain Trajectory Spatio-Temporal Representation via Distance-Preserving Triplet Learning
- HierLoc: Hyperbolic Entity Embeddings for Hierarchical Visual Geolocation
- Tile2Vec: Unsupervised Representation Learning for Spatially Distributed Data
- OmniSat: Self-supervised Modality Fusion for Earth Observation
- Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots
- Temporal Hallucinating for Action Recognition With Few Still Images
- Just a Glimpse: Rethinking Temporal Information for Video Continual Learning
- The "Something Something" Video Database for Learning and Evaluating Visual Common Sense
- Something-Else: Compositional Action Recognition With Spatial-Temporal Interaction Networks
- Action Tubelet Detector for Spatio-Temporal Action Localization