Hallucinating Motion at Every Scale: Per-Level Video Distillation for Action Recognition from Still Images

A single photograph of someone halfway out of a chair cannot say whether they are sitting down or standing up. Models asked to recognize actions from still images therefore fall back on what the picture does contain, such as the snow behind a skier or the sofa behind a person lying down. One established fix is temporal hallucination: a model learns from unlabeled video to predict, from a still image, the motion features a video model would have extracted had the clip been available. Existing hallucinators predict that motion as one global vector computed from the network's last layer. The multi-scale fusion literature shows that this last layer is exactly where fine detail disappears. Yet the cues a still image offers about motion are mostly fine-scale: blur on a moving hand, a limb leaning off balance, cloth trailing behind the body.

We propose to hallucinate motion separately at each level of a feature pyramid. A feature pyramid is a network that keeps feature maps at several resolutions and passes coarse semantic information down into the finer maps. Each pyramid level of an image model is trained to predict the matching level of a video teacher's motion pyramid, and the loss is computed only inside the actor's region. The testable claim is specific. With the input features and the teacher held fixed, per-level motion targets should beat a single global motion target for still-image action recognition. The gain should be largest on classes whose meaning depends on the direction of motion.

Core Claim and What Is New

Prior hallucination work supplies the setting and the strongest baselines. Temporal Hallucinating for Action Recognition With Few Still Images builds its Hybrid Video Memory by matching an image's global appearance against a bank of videos. A Gaussian process then blends the matched videos' optical-flow features into a hallucinated motion vector. Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition treats the still image as the starting state of a recurrent network that rolls out abstract motion embeddings. Watching Unlabeled Video Helps Learn New Human Actions from Very Few Labeled Snapshots extrapolates the poses just before and after the photographed moment on a pose manifold mined from video. All three produce one flat motion descriptor from one global embedding. Our proposal differs in where the motion is predicted, at every pyramid level, and in what it is regressed against, the teacher's motion features at the matching resolution.

The multi-scale literature motivates the design. Pixel-wise Deep Learning for Contour Detection finds that the deepest AlexNet layer alone reaches an ODS F-measure of 0.603 on BSDS500, while concatenating Conv1 through Conv5 reaches 0.741. CLIPCEIL finds that intermediate vision-transformer layers carry structure that generalizes better than the final layer. Two further papers supply specific techniques we reuse. Multiple Object Tracking via Feature Pyramid Siamese Networks contributes its learned top-down pathway and its contrastive loss applied at every pyramid level. That paper fuses its pyramid with a hand-crafted 12-dimensional motion vector late in the network and never learns motion features itself; we do the opposite. ST-HOI contributes the lesson that pooling a fixed region over a moving actor collects background and ruins temporal features. We therefore pool the teacher's motion targets along the actor's track.

Two papers constrain the proposal. Just a Glimpse reports that single frames are sufficient for most video classes in continual learning. This suggests that averaged over all classes, any motion signal will be small. We therefore report direction-dependent classes separately and do not claim uniform gains. The deeper constraint is physical. A still image does not determine motion direction, so hallucination can only recover the part of motion that leaves visible static traces. Part of the program is to measure how large that part is at each scale.

Research Program

The teacher is a two-stream video network trained on UCF101. It has an RGB stream and an optical-flow stream, and the flow stream is built as a feature pyramid with levels $\ell = 1,\dots,L$. For each training clip, we take the flow-stream feature maps around a center frame. We then pool them along the actor's box track, following ST-HOI's tube pooling, so that each level yields a motion descriptor $m_\ell$ tied to the actor rather than to the scene.

The student is an image pyramid with the same levels, applied only to the center frame. At each level, a small head $h_\ell$ reads the student's top-down-fused features, pooled in the actor box, and predicts $\hat m_\ell$. Training uses unlabeled video only:

$$\mathcal{L} = \sum_{\ell=1}^{L} \Big( \lVert \hat m_\ell - m_\ell \rVert^2 + \lambda\, \mathcal{L}^{\text{con}}_\ell \Big),$$

where $\mathcal{L}^{\text{con}}_\ell$ is an FPSN-style margin contrastive loss. It pulls $\hat m_\ell$ toward the motion descriptor of its own clip and pushes it away from other clips' descriptors at that level. The weight $\lambda$ balances the two terms. The contrastive term is there to stop the heads collapsing to a mean motion, which is a known failure when a regression target is ambiguous.

At downstream time on still images, the static pyramid features and the hallucinated $\hat m_1,\dots,\hat m_L$ are concatenated and fed to a linear classifier. The classifier is trained on few labeled images per class. Channel reduction keeps the concatenated descriptor from growing into the per-pixel bloat noted in the contour-detection work.

The analysis component asks one question: at which pyramid levels is motion predictable from a still image? We answer it with the cross-validated regression $R^2$ of each head, broken down by action class. We also test whether direction-dependent classes draw their decodable signal from the fine levels. If they do not, the per-level design has no reason to exist, and we would report that.

Baselines

The naive reference is the same image pyramid backbone, ImageNet-pretrained, with a linear classifier on its final pooled layer and no video data. A second static reference uses concatenated multi-scale static features without hallucination. It tells us how much of any gain comes merely from fine-scale inputs.

The strongest existing method is the Hybrid Video Memory of Temporal Hallucinating for Action Recognition With Few Still Images. It reports 35.4% accuracy on WEB101 with one image per class, against a 26.6% baseline. It also reports 42.2% against 39.0% on PASCAL VOC 2012, and 60.2% on the unseen-class DIFF20 set. The recurrent hallucinator of Exploiting Motion Information from Unlabeled Videos for Static Image Action Recognition is the second competitor. It reports 21.06% on PASCAL VOC and 18.76% on Stanford 40 Actions with ten labeled images per class, using UCF101 as the video source.

These published numbers come from older backbones and are cited only as anchors. For the actual comparison, we reimplement both methods on our backbone, with the same UCF101 video memory, the same labeled splits, and the same classifier. Each method gets the same hyperparameter search budget, measured in trials on a held-out split, and the same total training compute.

The ablation that isolates our key ingredient is a global-target control. It keeps our multi-scale student inputs and the same teacher. The only change is that the targets are pooled into one concatenated vector, predicted by a single head from the fused pyramid. Our proposal wins only if it beats this control by more than seed-to-seed variation. The control is run with five seeds, and the effect must hold on direction-dependent classes. A second ablation removes actor-region pooling of the targets. This tests whether whole-frame motion targets bring background motion back in as a shortcut.

Why not just…?

Why not just fuse the pyramid first and hallucinate one global motion vector? This is the real threat. If fine-scale motion cues can be carried through a fused global embedding, per-level targets add only complexity. We cannot rule it out from the literature, because no paper in the file runs the comparison. The global-target control above is built to settle it. It uses identical inputs and teacher and differs only in the structure of the target. If the control matches our method, we will report that the benefit comes from multi-scale inputs to hallucination, not from per-level supervision. That finding is still useful, because it would move existing hallucinators off the final layer.

Why not just use multi-scale static features with no video at all? Fine-scale static features may already encode blur and lean, which would make a video teacher redundant. The contour-detection results show that multi-scale features help localization, but they say nothing about motion. The static multi-scale baseline measures this directly. The per-class analysis then checks whether hallucination adds its gain on direction-dependent classes, where static cues should be weakest. A gain spread evenly across classes would instead suggest generic regularization.

Why not just feed the still image, repeated, into a pretrained video model? Repeating a frame to make a "boring video" worked well for continual learning in Just a Glimpse. But a repeated frame has zero frame-to-frame change, so a flow-based or 3D model sees no motion. ST-HOI also found that naive 3D backbones did worse than 2D models on temporal predicates. We include this baseline anyway, with the teacher's RGB stream on repeated frames. We expect it to match static baselines on scene-defined classes and to fall behind on direction-dependent ones.

Proposed Experiments

The main benchmarks are WEB101, PASCAL VOC 2012, DIFF20, and Stanford 40 Actions. Each is run with one, five, and ten labeled images per class. UCF101 serves as the unlabeled video source, and we swap in HMDB to test sensitivity to the source. The metric is mean accuracy over repeated random draws of the labeled images.

We add a direction diagnostic built from J-HMDB, which pairs classes such as sit and stand. Classification uses only the single middle frame of each clip, and we compare single-frame accuracy on these paired classes against scene-dominated classes. J-HMDB also gives an upper reference: the video-based Action Tubelet Detector reports 52.1% video-mAP at IoU 0.75 with six-frame tubelets. That number is for a different task, localization rather than classification, and serves only to bound what full temporal context buys.

The decisive checks are three. First, the per-level method must beat the global-target control by more than seed variation on the direction diagnostic, with the same backbone and budget. Second, fine-level heads must show higher cross-validated motion $R^2$ than coarse heads on direction-dependent classes. Third, removing actor-region pooling must increase reliance on background, measured by accuracy loss when the actor is masked out versus when the background is masked out. If the first check fails, the per-level design is not worth doing, and the note's contribution reduces to the scale-by-scale measurement of how much motion a still image reveals.