When a face blurs, turns away, or is covered by a hand for half a second, current video models lose more than a few frames. They lose whatever the face did during that time. Methods that borrow features from neighbouring frames can, at best, interpolate between the last clean frame and the next one. So an expression that begins and ends inside the corrupted window, such as a quick laugh, a flinch, or a frown that relaxes, simply never happened as far as the model can tell. The voice usually keeps going through these gaps. Our proposal is to train, without labels, a model that uses the audio heard during a visual gap to predict what the face was doing frame by frame inside that gap. It anchors this prediction on the clean frames on either side.
The claim is narrow and testable. Audio is already known to help emotion classification, and a model that falls back to audio when vision fails will recover much of a clip-level label. Our claim is different. Audio tied to the visual states at the gap boundaries can recover the shape of the expression trajectory inside the gap: when the apex happened, and whether the face left neutral and came back. Neither visual interpolation nor audio-only fallback can do this. Interpolation has no information about the inside of the gap. Audio alone has no anchor to this particular face's resting state and current expression. If the claim holds, the gain should be concentrated on "in-gap events", meaning expressions that both start and end inside a degraded window. On clip-level accuracy the gain should be small. That pattern is the signature we will look for.
What exists and where this sits
Three nearby lines of work each cover part of this problem.
AVF-MAE++ pre-trains audio-visual masked autoencoders on 1.36M face clips and reports above 60% weighted average recall on 11-class MAFW. A masked autoencoder is a network trained to fill in hidden patches. AVF-MAE++ hides random scattered patches. It does not hide temporally contiguous stretches that resemble real blur or occlusion, and it does not evaluate under physical degradation. This paper supplies our backbone and is our strongest competitor.
AVLR shows the core intuition works for speech. It uses synchronized audio to re-synthesize occluded lips and brings AV-HuBERT word error at −5dB from 13.87% back to 11.87% (clean: 10.62%). However, it restores pixels, it is supervised by a phonetic loss, it covers only the mouth, and it requires one uncorrupted reference frame. It motivates our proposal and also warns us: the authors report hallucinated lip motion when audio is poor.
MMIN "imagines" a missing modality from the available ones, but it works on pooled sentence-level vectors with whole modalities zeroed out. It never models a transient gap within a frame sequence.
On the vision side, FGFA and STSN recover degraded frames by sampling features from sharp neighbours. FGFA raises fast-motion ImageNet VID mAP from 51.4% to 57.6%. Both methods assume that clean neighbours exist and contain the answer. DP-TempCoh restores face video with temporal coherence (IFD 3.80 vs. CodeFormer's 9.86) but is vision-only and indifferent to affect. These methods constrain our design. They are the right baselines for the case where the gap is short and the expression is changing slowly, and our method has to beat them where that assumption breaks.
From the expression-dynamics literature we take the reason trajectory shape matters. Averaging over a clip makes neutral-to-anger indistinguishable from anger-to-neutral (Jiang et al.). Onset dynamics separate spontaneous from posed expressions (Bartlett et al.). MAFW's snippet-order shuffling (T-ESFL, 48.18% WAR) shows that forcing a model to reason about temporal order helps. Our negative examples make a similar demand, but across modalities.
Core mechanism
We start with a pretrained audio-visual backbone and freeze a copy of its visual encoder to act as a teacher. For each training clip we corrupt a contiguous window of $L$ frames, with $L$ drawn from 4 to 32 frames. The corruption is physically motivated: directional motion blur, defocus, synthetic hand or object occlusion overlays, or full frame dropout. A student network sees three inputs: the clean frames before the window, the clean frames after it, the corrupted frames, and the full audio track. It outputs a predicted latent trajectory $\hat z_{1:L}$ for the window. The target is the teacher's latents $z_{1:L}$ computed on the uncorrupted frames. Because the corruption is synthetic, the clean target is free, and the whole procedure is self-supervised.
Regressing toward the target alone would let the student learn smooth interpolation between the boundaries and ignore the audio. To prevent this, we train with a contrastive objective. Contrastive here means scoring the true answer against hard alternatives. The alternatives are built to share the same boundaries as the true trajectory, so they differ only in what happens inside the window:
$$ \mathcal{L} = -\log \frac{\exp(s(\hat z, z)/\tau)}{\exp(s(\hat z, z)/\tau) + \sum_{n \in \mathcal{N}} \exp(s(\hat z, z^{(n)})/\tau)} $$
Here $s$ is the mean per-frame cosine similarity between two latent trajectories and $\tau$ is a temperature. $\mathcal{N}$ is a set of negative trajectories made from the same clip, of three kinds. The first is the true in-window trajectory reversed in time, which swaps onset and decay. The second is the true trajectory time-warped so that its apex moves within the window. The third is the linear interpolation between the boundary latents, which corresponds to "nothing happened". All three are consistent with the visible boundaries, so the student can separate them only by reading the timing in the audio. This is the step that goes beyond the uniform ±k-frame audio shift used in Audio-Visual Synchronisation in the wild: the negatives probe expression phase rather than raw synchrony. A small regression term toward $z$ is kept for stability.
The student also predicts a per-frame confidence. On silent stretches, such as a smile with no vocalization, the correct behaviour is to fall back toward interpolation and report low confidence, not to invent motion. We measure this directly because of the hallucination failure AVLR reports.
At test time the student replaces the visual features inside detected degraded windows. Any downstream expression head consumes the result unchanged. Degraded windows are detected with a simple blur and occlusion detector; we also report results with oracle windows.
Research Program
The work has three stages. First, we pre-train the student on the unlabeled face corpora used by AVF-MAE++ (VoxCeleb2, CN-Celeb, AV-Speech), starting from AVF-MAE++ weights so that any gain cannot be attributed to more pre-training data. Second, we evaluate frozen and fine-tuned students on degraded versions of labeled benchmarks. Third, we run a diagnostic study that splits results by gap length, by whether the gap contains vocalization, and by whether an expression event lies wholly inside the gap.
The diagnostic study is the heart of the proposal. It is designed to show exactly where audio-anchored recovery helps and where it should not. On Aff-Wild2, whose per-frame valence and arousal labels make this possible, we locate excursions that leave and return to baseline within 32 frames. We then place corruption windows that fully cover them ("in-gap events"). We compare these against windows placed over steady segments.
Baselines
The do-nothing reference is the strongest clean-trained expression model (AVF-MAE++ fine-tuned on clean data) run directly on degraded test video. This shows how much degradation costs. On clean data, the reference points from the file are VGGFace1-GRU-attention on Aff-Wild2 at CCC 0.55 valence and 0.45 arousal (Aff-Wild2), and AVF-MAE++ above 60% WAR on MAFW. We found no reported result on Lacuna for any of these benchmarks under contiguous synthetic visual degradation, so all degraded-condition numbers will be produced by us under one protocol.
The strongest competitor is AVF-MAE++ fine-tuned with the same degradation augmentation we use. We give it the same unlabeled pre-training corpus, a continued-pretraining phase that hides contiguous windows instead of random patches, and the same fine-tuning compute and hyperparameter search budget as our student. This is the cheapest way to "just make the existing model robust". Our method beats it only if it improves in-gap-event CCC by a margin larger than seed variance, measured over three seeds, while matching it on clip-level WAR.
The visual-only temporal competitor is an STSN-style deformable sampler (STSN) over the AVF-MAE++ visual features, trained on the same degraded data. A latent linear-interpolation baseline is the trivial version of the same idea.
The audio-fallback competitor is a gated fusion model. It shares our backbone and down-weights visual features inside detected degraded windows, so the prediction there rests on audio alone. A frame-level adaptation of MMIN's imagination module, applied per window rather than per sentence, is the published version of this idea.
The two-stage competitor runs face video restoration first and recognition second. We use DP-TempCoh if code is available and CodeFormer otherwise.
Two ablations isolate the key ingredients. In the first, we train the same student with regression only, without the phase-shuffled negatives. In the second, we keep the negatives but replace them with uniform audio time-shifts. If either ablation matches the full model, the phase-aware negatives are not doing the work.
Why not just…?
Why not just fine-tune AVF-MAE++ with blur and occlusion augmentation? This is the most serious threat, and we treat it as the primary baseline rather than arguing it away. Its masked-reconstruction objective already uses audio. With contiguous masking it might learn gap bridging on its own. Our hypothesis is that reconstruction losses reward the smooth average of plausible in-gap trajectories, which on an expression that leaves neutral and returns is close to the interpolation. The phase-shuffled negatives penalize exactly that average. The regression-only ablation tests this directly. If AVF-MAE++ with contiguous masking matches us on in-gap events, we will report that the objective change is unnecessary and that the contribution reduces to an evaluation protocol.
Why not just fall back to audio when the face is unreadable? For clip-level labels this probably works almost as well as our method. Prosody carries category information, and MAFW reports a 3.26% WAR gain from adding audio and text. Where it should fail is per-frame tracking. Audio does not tell you this person's resting valence, their current expression, or how quickly their face returns to neutral, and those come from the boundary frames. The gated-fusion baseline is evaluated on in-gap CCC specifically. If it matches us there, the boundary anchoring adds nothing and the proposal is wrong.
Why not just aggregate sharp neighbouring frames as FGFA or STSN do? These methods work for objects that persist through blur. They cannot recover an event that happens entirely while the view is degraded, because no neighbouring frame contains it. We expect them to match or beat us on short gaps over steady segments, and to fall to interpolation level on in-gap events. The gap-length breakdown will show where the crossover happens.
Why not just restore the frames first and then classify? Restoration optimizes appearance rather than expression timing. Frame-wise restorers flicker (CodeFormer IFD 9.86 in DP-TempCoh), and flicker reads as spurious facial motion. In addition, a vision-only restorer under full occlusion has nothing to restore from. The two-stage baseline will be scored on the same in-gap metrics as everything else.
Proposed Experiments
Degraded benchmarks will be built from DFEW and MAFW (clip-level 7- and 11-class WAR and unweighted average recall), Aff-Wild2 (per-frame valence and arousal CCC), and IEMOCAP for a missing-modality comparison with MMIN under frame-level gaps. Corruption types, window lengths, and window placements are fixed and released, so that every method sees identical inputs.
The decisive check is the Aff-Wild2 in-gap-event split. Success means our method's CCC on fully covered excursions clearly exceeds that of the interpolation, STSN-style, gated-audio, and augmented AVF-MAE++ baselines, while clip-level WAR on MAFW and DFEW stays within noise of the augmented AVF-MAE++ baseline.
The second check is silence. On in-gap windows with no vocal activity, our predicted trajectories should not be worse than interpolation, and the confidence output should drop. If trajectories in silent windows deviate from interpolation while confidence stays high, the model is hallucinating, and we will report it as such.
The third check is phase specificity. Given true versus time-reversed visual trajectories for held-out gaps, a probe on the student should pick the correct one well above chance only when audio is present. If it succeeds with the audio replaced by audio from another clip, the model is exploiting visual leakage rather than cross-modal timing.
References
- AVF-MAE++: Scaling Affective Video Facial Masked Autoencoders via Efficient Audio-Visual Self-Supervised Learning
- Restoring Speaking Lips from Occlusion for Audio-Visual Speech Recognition
- Missing Modality Imagination Network for Emotion Recognition with Uncertain Missing Modalities
- Flow-Guided Feature Aggregation for Video Object Detection
- Object Detection in Video with Spatiotemporal Sampling Networks
- Discrete Prior-Based Temporal-Coherent Content Prediction for Blind Face Video Restoration
- Predicting Emotions in User-Generated Videos
- A Prototype for Automatic Recognition of Spontaneous Facial Actions
- MAFW: A Large-scale, Multi-modal, Compound Affective Database for Dynamic Facial Expression Recognition in the Wild
- Audio-Visual Synchronisation in the wild
- Aff-Wild2: Extending the Aff-Wild Database for Affect Recognition