Local-in-Time Attention Gates for Lightweight In-the-Wild Affect Video

Models that read continuous emotion from face video "in the wild" (YouTube clips with bad lighting, turned heads, hands over mouths) are built on heavy image networks such as ResNet-50 or VGGFace. These networks shrink each frame to a single feature vector, and a recurrent or temporal-convolution head then smooths that vector over time. Efficient video networks are a separate line of work. They cut the cost of 3D convolution by factorizing it or by shifting channels in time, and they have been tested only on action recognition. We propose that when a thin network is used for affect, most of its robustness should come from a specific kind of attention. For every feature channel, at every spatial location, the network decides how much to trust that location given what the same location showed a few frames earlier. The gate is built entirely from depthwise convolutions, meaning each channel is filtered on its own, and it therefore costs almost nothing.

The argument is mechanistic. Most nuisances in the wild are local in space and brief in time: a shadow crosses one cheek, or a hand covers the mouth for half a second. A temporal head that sees only globally pooled vectors cannot tell that the mouth region in particular became unreliable. A gate that works on the spatial map and has a short temporal receptive field can down-weight that region and let neighboring frames carry the signal. If the idea is right, a thin X3D-style network with this gate should lose far less accuracy under localized, transient corruption than the same network with standard squeeze-and-excitation plus a recurrent head. It should also approach heavy two-stream ResNet-50 systems on Aff-Wild2 while running within real-time budgets on an embedded GPU.

Core Claim

The gate operates on a feature tensor $X$ with $C$ channels, $T$ frames and $H \times W$ spatial positions. It is computed as

$$G = \sigma\big(D_t(D_s(X))\big), \qquad Y = X \odot G,$$

Here $D_s$ is a depthwise $k_s \times k_s$ spatial convolution (two layers, as in the squeeze step of Depthwise Squeeze and Refinement), $D_t$ is a depthwise causal temporal convolution of length $k_t$, $\sigma$ is the sigmoid, and $\odot$ is elementwise multiplication. The gate costs about $C(2k_s^2 + k_t)$ multiply-adds per position. A squeeze-and-excitation (SE) block, by contrast, spends $O(C^2)$ on dense layers after pooling away all spatial layout.

The claim has three parts, and each is falsifiable. First, the gate's benefit on Aff-Wild2 should concentrate in corrupted windows rather than spread uniformly. A uniform gain would indicate extra capacity, not the proposed mechanism. Second, the temporal kernel matters: setting $k_t=1$ reduces the gate to a video-applied version of the static-image DSR block, and that version should lose most of the robustness gain. Third, the thin-network question raised by Hassani et al. has a definite answer. Hassani et al. argue that deep, high-capacity residual networks are needed to absorb low-level clutter in wild video. X3D shows that thin, high-resolution separable networks win on action recognition. We predict that thinness is not the problem; losing spatial locality before temporal integration is.

Relation to Prior Work

The gate reuses techniques from three static-image papers. DSR replaces SE's global pooling with parameterized depthwise convolutions and drops cross-channel mixing, improving over SE on ImageNet32 (58.22% vs 55.87% top-1 for ResNet-50). Light-MUSTER replaces the dense key and value projections in a transformer decoder with depthwise convolutions and loses almost no ADE20K accuracy (50.18 vs 50.23 mIoU). DE-CBAM folds depthwise separable convolutions into CBAM for a 1.9M-parameter retinal segmenter. None of the three has a time axis, and none is tested under nuisance shift. We add the temporal depthwise stage and evaluate specifically under corruption.

Two of these papers also constrain the design. MUSTER reports jagged boundaries when dense projections become depthwise. DE-CBAM reports 2.04 s per image on CPU despite its tiny parameter count, because multi-branch depthwise structures fragment memory access. We therefore keep the gate single-branch, measure wall-clock latency rather than FLOPs alone, and treat any latency regression as a result to report.

The video efficiency literature supplies the host backbones and the competitors. X3D grows a channel-separable 2D base into a 3D network; X3D-XS runs at 0.6 GFLOPs. TSM shifts a quarter of the channels in time at zero FLOPs and reaches 13 ms on a Jetson Nano. FstCN factorizes 3D kernels into 2D spatial and 1D temporal parts. Ada3D routes clips between 2D and 3D kernels but admits that branching erodes its real speedups. All four were evaluated on action datasets, where the label depends on coarse motion, not on subtle facial muscle changes under corruption.

The affect literature motivates the work. Aff-Wild2, multi-task ArcFace models, leader-follower TCN fusion and the ABAW2 phase-difference report all pool spatially before temporal modeling, and all use ResNet- or VGG-scale backbones. None of them reports edge latency.

The file raises a tension: DSR's removal of channel mixing may conflict with affect's apparent need for cross-feature interaction. We test this directly with a variant that adds a cheap grouped pointwise mix inside the gate.

Research Program

We will build the gate as a drop-in replacement for the SE blocks inside X3D-XS and X3D-S. We will also insert it after the shift in a TSM-MobileNetV2. In both hosts, the global temporal head is reduced to a small causal depthwise temporal convolution over pooled features, so that long-range smoothing is not doing the work the gate is meant to do. Training uses Aff-Wild2 valence-arousal with a CCC loss (CCC, the concordance correlation coefficient, measures both correlation and agreement in scale with the continuous labels). Label distribution smoothing is adopted from the ABAW2 report, so our imbalance handling matches theirs. Backbones are pre-trained on Kinetics-400, with an AffectNet-pretrained variant as an alternative.

For analysis, we will build a corruption protocol on the Aff-Wild2 validation split. Face-aligned crops receive occluders over landmark-defined eye or mouth regions, or synthetic shadow and gamma shifts over half the face. Each corruption lasts a random window of 5 to 30 frames, and a global-corruption control applies the same corruptions to the whole face or the whole clip. We also stratify natural frames by off-the-shelf head-pose yaw and by face-detector confidence. Throughout, we will visualize gate maps to check whether the gate actually suppresses corrupted regions.

For deployment, we will profile a Jetson Nano (the device TSM reports on) and a CPU, with the gate fused into the host kernels where possible.

Baselines

The naive reference is the same X3D-XS backbone run per clip, with a linear valence-arousal readout, no attention, and no temporal head. It shows what the thin network does on its own.

The strongest existing method on this question is the two-stream ResNet-50 with phase-difference images from the ABAW2 report. On the Aff-Wild2 validation split it reports valence CCC 0.591, arousal CCC 0.617, and mean CCC 0.560 with label smoothing, reported in the file as 0.552 without it. The original Aff-Wild2 CNN-GRU reports 0.55 valence and 0.45 arousal CCC. It remains the canonical pooled-then-recurrent design, so we will reimplement it as a second heavy baseline. Because splits and preprocessing differ across papers, we will retrain both heavy models under our pipeline: same face crops, same epochs, same hyperparameter search budget. We will report both our reproduced numbers and the published ones. For latency, there is no reported edge-device result on Lacuna for any Aff-Wild2 model, so we measure all baselines ourselves.

The strongest efficient competitors are X3D-XS fine-tuned as-is (with its native SE blocks) plus a two-layer GRU head, and TSM-MobileNetV2 plus a GRU head. Each gets the same tuning budget as our model.

Ablations isolate the key ingredient. The first sets $k_t=1$ (static DSR on video). The second replaces the spatial depthwise squeeze with global pooling while keeping the temporal kernel (a temporal SE). The third adds grouped pointwise channel mixing inside the gate. The fourth matches the gate's parameter count by widening the backbone instead of adding the gate.

Beating the baselines means the following. We must match the retrained two-stream ResNet-50 within a small CCC margin on clean validation data at a fraction of its latency. We must also show a significantly smaller CCC drop than the SE+GRU thin baselines on localized, transient corruptions, with no such advantage under global corruption.

Why not just…?

Why not just fine-tune X3D, which already contains depthwise convolutions and SE blocks? This is the most serious threat. X3D's SE pools globally, so by our hypothesis it cannot localize a transient occluder, but that is a prediction, not evidence. We run X3D-XS unmodified with and without a GRU head under the identical protocol. If it matches our gate on the corruption benchmark, the proposal's mechanism is refuted, and we would report that thin off-the-shelf video networks already suffice for wild affect. That would itself be a new finding given the Hassani et al. position.

Why not just use TSM with a recurrent head? TSM is cheaper still and has proven edge latency. Its shift, however, is fixed and blind to content: it mixes neighboring frames equally whether or not the current mouth region is occluded. The TSM+GRU baseline and the TSM+gate variant use the same host, so the comparison isolates whether content-dependent, per-location trust adds anything beyond fixed temporal mixing. If the gain is within noise, we will conclude that the shift already carries enough temporal context and that the gate is not worth its latency.

Why not just distill a heavy ResNet-50 affect model into a small student? Distillation targets the capacity gap, not the pooling-before-time structure, so the two ideas are complementary. It could nonetheless close the clean-accuracy gap by itself. We therefore distill from the retrained two-stream teacher into both the SE+GRU student and the gated student. If distillation erases the gated model's corruption advantage, the gate is unnecessary. If the advantage survives distillation, the gate adds robustness that distillation alone does not.

Proposed Experiments

The first decisive check is the stratified CCC comparison. The gated thin model should beat SE+GRU thin baselines inside corrupted windows, stay roughly equal outside them, and show no advantage under full-face corruption. That pattern is what separates the locality mechanism from generic extra capacity.

The second check is the $k_t$ sweep (1, 3, 5, 9 frames). We expect the robustness gain to rise and then saturate near the typical duration of an occlusion, and to be near zero at $k_t=1$. The channel-mixing ablation tests the DSR tension: if grouped mixing helps valence more than arousal, that would echo the modality asymmetry observed in the leader-follower work.

Third, we report clean Aff-Wild2 validation CCC against the retrained ABAW2 two-stream model, and transfer to RECOLA following the Aff-Wild2 pretraining protocol.

Finally, we report Jetson Nano and CPU latency per frame for every model, alongside FLOPs. Given the DE-CBAM and Ada3D experience, we treat a FLOP reduction without a wall-clock reduction as a negative result rather than a success.