When a person in a video is reduced to a few dozen pixels, as in distant surveillance footage or a clip downscaled to 12×16, the usual ways of learning video features without labels stop working. One common label-free task asks a network to put shuffled clips back in order. The network can usually solve this from things that have nothing to do with the action, such as background drift, compression noise, or interpolation artifacts. At extreme low resolution those artifacts make up almost the whole image. Audio does not lose detail when the picture shrinks, and the sounds in a clip are mostly made by the things that move in it. We propose to use the clip's own soundtrack as a clock. The network sees the audio of a clip in its true order and must sort the shuffled, pixel-starved video segments to match it. It does this through cross-modal attention, a mechanism in which audio features "query" the spatial positions of the video features and weight them by relevance.
There are two consequences. First, every segment comes from the same clip, so scene-level cues like "beach sounds go with beach pictures" carry no information about order. The only way to solve the task is to find the small moving thing whose motion lines up with the sound. Second, where the audio queries choose to look is a free, label-free estimate of where the actor is. We distill that map into a learnable pooling layer inside the video encoder, so that at test time the encoder runs on video alone and still keeps the few pixels that matter. Prior methods for extreme low-resolution video depend on labels and a high-resolution teacher. This approach needs neither.
Core Claim
Within-clip, audio-anchored temporal ordering is a self-supervised task that cannot be solved from static background or downsampling artifacts. Its cross-modal attention map provides foreground localization that existing methods buy with supervision. The specific prediction is that a video encoder pretrained this way on unlabeled, synthetically degraded audio-video data will beat two alternatives on video-only action recognition at 12×16. The first is the same encoder pretrained with video-only sorting. The second is the same encoder pretrained with clip-level audio-visual contrastive learning. The gain should come from two ingredients that can be removed one at a time: the within-clip ordering design and the attention-to-pooling transfer.
What Prior Work Does and Leaves Open
The closest existing work is Audio-Visual Contrastive Learning with Temporal Self-Supervision. It combines audio-video contrastive alignment with playback speed and direction prediction and a cross-modal "is this audio-video pair aligned, misordered, or overlapping" task. It supplies two pieces we reuse: the idea of cross-modal temporal ordering, and its finding that raw waveforms beat spectrograms for temporal pretext tasks. It differs from us in three ways. It fuses global clip embeddings rather than letting audio attend over spatial positions, so it produces no localization map. Its negatives span different clips, so scene identity remains a usable cue. It is evaluated only on clean, full-resolution video.
Two lines of work motivate why plain video sorting needs help in our regime. Unsupervised Representation Learning by Sorting Sequences introduced the Order Prediction Network and documented that networks sort frames using chromatic aberration and background cues. Its countermeasures, spatial jitter, cropping, and channel splitting, would erase a 10×10-pixel actor. DirecFormer adds an order-prediction loss that pushes attention to follow chronology, but it is tested on high-resolution single-modality video.
On the degradation side, two papers both rely on supervision to find the actor. Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer, which we call CSTAT, reaches strong 12×16 accuracy by distilling attention from a labeled high-resolution teacher. TinyVIRAT pairs heavy video super-resolution with weakly supervised foreground attention. Our audio map is meant to replace both the teacher and the labels.
CSTAT also constrains the design. It shows that forcing a degraded stream to match attention from a clean source hurts when the attended detail has vanished, and that confidence weighting is needed. An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention shows the related failure in fusion, where one rich stream swamps the other unless a learned gate controls the mix. We therefore gate every audio query, so that silent or off-screen sound segments can opt out instead of injecting noise.
The pooling itself comes from LIP: Local Importance-Based Pooling. LIP learns a per-location importance map for downsampling and reports its largest gains on small objects. In LIP the map is learned only from the end task. We supervise it with the audio attention.
Research Program
The model has two encoders. The visual encoder is a 3D convolutional network matched to CSTAT's backbones for comparability. It runs on 12×16 frames degraded with CSTAT's protocol of average downsampling, lens blur, and Gaussian noise, and every strided downsampling layer is replaced by a LIP-style importance pooling layer. The audio encoder runs on raw waveform, following the waveform finding of the closest prior work.
Each training clip is split into four consecutive segments. The video segments are shuffled. The audio segments stay in order and are encoded with continuous time encodings, so that audio and video at different rates share one time axis, as in Temporal and Cross-modal Attention for Audio-Visual Zero-Shot Learning. Audio tokens act as queries over the space-time tokens of the shuffled video segments. A gate $g \in [0,1]$ is predicted from each audio token and scales its contribution, in the form output $= g \cdot$ cross-attended context $+ (1-g) \cdot$ original video features. A head then predicts which of the 24 permutations was applied. The audio anchor fixes the true direction, so no symmetry folding is needed, unlike the Order Prediction Network.
The gated, averaged cross-attention over spatial positions gives a map $A_s$ for each video segment. We train the pooling importance logits $\ell_s$ of the visual encoder to match it:
$$ \mathcal{L}_{\text{pool}} = \sum_{s} \bar{g}_s \, \mathrm{KL}\!\left(\mathrm{sg}(A_s) \,\|\, \mathrm{softmax}(\ell_s)\right), $$
where $s$ indexes video segments, $\bar{g}_s$ is the mean gate value of the audio tokens attending to segment $s$, and $\mathrm{sg}$ stops gradients into the attention. The gate weighting is how CSTAT's lesson about confidence enters the design: segments whose sound does not ground anything visible do not teach the pooling where to look. The total loss is the permutation cross-entropy plus $\lambda \mathcal{L}_{\text{pool}}$. After pretraining, the audio branch is discarded, and downstream evaluation uses the visual encoder alone.
The analysis track asks whether the map really localizes the actor. Synthetic degradation keeps the original high-resolution frames. These are never used in training, but at evaluation they let us compare $A_s$ against foreground regions derived from the high-resolution frames.
Baselines
The naive reference point is the same visual encoder trained only on labeled 12×16 data, with standard strided pooling and no pretraining. CSTAT reports this student baseline at 42.79% on UCF101 at 12×16 with a ResNet-34 backbone. We rerun it under our settings rather than relying only on that figure.
The second baseline is video-only temporal sorting: the Order Prediction Network objective applied to the degraded clips with the same encoder and pretraining data. This isolates whether audio is needed at all. For context, the Order Prediction Network paper reports 56.3% on UCF101 at full resolution. That figure is not comparable to 12×16, and no reported result was found on Lacuna for sorting-based pretraining at extreme low resolution.
The strongest self-supervised competitor is the recipe from the closest prior work, clip-level audio-video contrastive learning plus its temporal tasks, re-implemented on degraded video. It is the strongest because it already uses audio and ordering. Its reported 91.8% on UCF101 is at full resolution, and no reported result was found on Lacuna for it under spatial degradation.
The strongest overall reference is CSTAT, which uses labels and a high-resolution teacher. It reports 59.41% on HMDB51 at 12×16 with R2Plus1D-34, and 50.78% on UCF101 at 12×16 for its ResNet-34 student with rolling weights and unlabeled data expansion. We treat CSTAT as a ceiling to approach, not a method we must beat, since it consumes supervision ours does not.
Ablations isolate the proposal's key ingredients. The first keeps audio-anchored ordering but uses uniform pooling and drops $\mathcal{L}_{\text{pool}}$. The second keeps everything but removes the gate. The third replaces within-clip ordering with cross-clip audio-video correspondence, holding the architecture fixed. The fourth pairs the video with audio from a different clip, which should collapse ordering to chance and verifies that no video-only shortcut remains.
Fairness rules apply to all self-supervised variants. They share the backbone, the pretraining set (Kinetics-400 clips with audio, degraded to 12×16), the number of pretraining epochs, the augmentations that are safe at this resolution, and an equal hyperparameter search budget. Fine-tuning and linear-probe protocols on HMDB51 and UCF101 follow CSTAT's degradation and split conventions.
Beating the strongest competitor means two things on both HMDB51 and UCF101 at 12×16. Our method must outperform the degraded re-implementation of the closest prior work by more than the seed-to-seed spread across three seeds. It must also do so in both linear probing and full fine-tuning.
Why not just…?
Why not just run the existing audio-visual contrastive recipe on degraded video? This is the real threat, and the main experiment exists to settle it. Our argument against it is mechanistic. Its negatives come from other clips, and its alignment tasks can be solved with clip-level embeddings, so a scene-level match between background appearance and ambient sound is a sufficient strategy. That strategy gets easier, not harder, when the actor disappears into a few pixels. The cross-clip correspondence ablation tests exactly this, since it holds our architecture fixed and swaps only the task. If it matches the full model, the within-clip claim is wrong, and the contribution reduces to the attention-to-pooling transfer.
Why not just pretrain on high-resolution video and distill into the low-resolution encoder? When high-resolution versions exist, this is a strong and simple option. CSTAT shows that teacher attention transfer works with confidence weighting, and a self-supervised teacher would remove its need for labels. We will run it: a self-supervised high-resolution teacher distilled into the 12×16 student on the same data. If it wins, we will say so, and the method's value narrows to footage that is natively low-resolution, where no teacher input exists. We then test that setting by transferring our audio-free encoder to TinyVIRAT. The file describes TinyVIRAT as surveillance footage and does not describe audio, so this is a video-only transfer test. Its reported best F1 is 34.49, from the TinyVIRAT paper.
Why not just super-resolve the frames first? TinyVIRAT already shows the limit of this. Even with 3D super-resolution, F1 sits at 28.73 until foreground attention is added, because uniform enhancement sharpens background clutter as readily as the actor. Super-resolution also adds the heavy compute that the low-resolution setting usually cannot afford. We include a bicubic-upsampling input variant of video-only sorting as a cheap check.
Why not just use learnable pooling without audio? LIP alone learns importance only from whatever loss is applied. Under video-only sorting, that loss rewards attending to the very background artifacts that solve the shortcut. The uniform-pooling ablation and the comparison with LIP plus video-only sorting separate these effects.
Proposed Experiments
The decisive comparison is pretraining on degraded Kinetics-400 audio-video followed by linear probing and fine-tuning on HMDB51 and UCF101 at 12×16, against all baselines above. The full method should beat both video-only sorting and the degraded audio-visual contrastive recipe. Removing either the within-clip design or $\mathcal{L}_{\text{pool}}$ should erase most of the gain.
Two shortcut diagnostics follow. In a static control, each video segment is replaced with a repeated single frame plus fresh noise; ordering accuracy must fall to chance. Mismatched audio must also drive ordering to chance.
The localization check scores the gated attention map against foreground masks taken from the held-out high-resolution frames. Scores should rise during pretraining and should track the per-clip gate value. If the gate stays high on clips whose sound comes from off-screen sources, the confidence mechanism is not working.
Transfer runs the audio-free encoder on TinyVIRAT, comparing against the reported I3D F1 of 28.73 and full-model F1 of 34.49 from the TinyVIRAT paper. We also report audio-visual classification on VGGSound with degraded frames, alongside the reported clean-input supervised ResNet-50 top-1 of 51.0% from VGGSound, to measure how much of the clean-resolution performance survives.
A negative result on the main comparison would still be informative. It would mean that, at this resolution, scene-level audio-visual agreement already carries most of the learnable signal.
References
- Audio-Visual Contrastive Learning with Temporal Self-Supervision
- Unsupervised Representation Learning by Sorting Sequences
- DirecFormer: A Directed Attention in Transformer Approach to Robust Action Recognition
- Extreme Low-Resolution Action Recognition with Confident Spatial-Temporal Attention Transfer
- TinyVIRAT: Low-resolution Video Action Recognition
- An Interpretable Framework for Drug-Target Interaction with Gated Cross Attention
- LIP: Local Importance-Based Pooling
- Temporal and Cross-modal Attention for Audio-Visual Zero-Shot Learning
- Vggsound: A Large-Scale Audio-Visual Dataset