Emotion-recognition models are usually trained and tested on video that is assumed to be clean. Real deployments are different: video calls, streaming clips and phone recordings arrive heavily compressed, and the compression damage changes from frame to frame and from region to region. The video encoder already knows where the damage is. For every block of every frame it records how coarsely the block was quantized (the quantization parameter, QP), whether the frame was coded on its own (an I-frame) or predicted from neighbours (a P- or B-frame), how large the prediction error was, and how the block moved. This information comes free with the bitstream and is exact. We propose to feed it to a small, edge-deployable temporal affect model as a per-frame and per-region reliability signal. The model would then learn to rely on well-coded moments and regions and to bridge poorly coded ones from temporal context, rather than trying to infer the damage from the pixels.
The work would be built on Aff-Wild2, the YouTube-sourced benchmark for continuous valence and arousal, facial action units and expressions. It would use a backbone built from depthwise separable convolutions, which split a standard convolution into a per-channel spatial filter followed by a 1×1 channel mixer and remove roughly 90% of the operations. The question the work answers is concrete. With an equally strong compression-augmented model as the control, does explicit codec reliability improve accuracy and temporal stability under heavy quantization? A secondary question is whether the cheap factorized backbone keeps that robustness or loses it.
Core Claim
In compressed in-the-wild video, affect-recognition errors should rise and fall with codec-reported quality: QP, frame type and residual energy. A model that conditions on these signals should beat an otherwise identical model trained only with compression augmentation. Two conditions apply. The gain should grow with QP. The gain should also not be reproducible by a reliability estimate learned from the decoded pixels. If a pixel-based quality estimator matches the codec signal, the bitstream adds nothing, and the honest conclusion is "quality-aware temporal weighting" rather than "codec-aware". The experiments are designed so that either outcome is informative.
Relation to Prior Work
Two papers motivate the gap. Identity-aware Facial Expression Recognition in Compressed Video recognizes expressions directly from H.264 I-frames and P-frame residuals with a standard CNN and LSTM. It reaches 52.01% on AFEW and runs at 35 fps, against 11 fps for pipelines that fully decode the video. It treats the bitstream as a cheaper input, not as a measure of input quality. It also reports that motion vectors were too coarse for expression cues. We do not try to recover motion from the vectors. We use QP and residual statistics as confidence, which is a different role. Optimized Learned Image Compression for Facial Expression Recognition shows that task-aware compression preserves expression cues in single AffectNet images at about 0.1 bits per pixel. That approach controls the encoder. We assume a standard codec we cannot change and work only on the receiving side, over time.
Affective Processes supplies the temporal method. It is a roughly 330,000-parameter neural process that builds a global context from low-uncertainty frames and produces smooth valence and arousal trajectories, with CCC 0.75 and 0.64 on SEWA. It was evaluated only on uncompressed frames, and its context selection relies on the model's own uncertainty. That uncertainty is exactly what compression may corrupt. We add codec reliability as an input to this context selection.
The restoration literature sets constraints rather than providing the method. CPGA, MF-DLUT and RivuletMLP show that bitstream priors (motion vectors, predictive and residual frames) replace optical flow cheaply in pixel restoration. All three optimize PSNR on generic sequences and never measure downstream perception. Sensitivity Decouple Learning separates compression-insensitive semantics from a learned degradation-degree cue, lifting COCO detection on heavily compressed JPEGs from 9.5% to 26.1% mAP. Its learned degradation cue is the pixel-based rival to our codec signal, and we reuse it as an ablation.
For efficiency, Xception provides the factorized building block. GDWS shows that a trained standard convolution can be factorized after training into generalized depthwise separable form while keeping adversarial robustness. On a Jetson Xavier this took a CIFAR-10 ResNet-18 from 28 to 104 FPS. Whether that preservation extends to structured compression damage, rather than norm-bounded noise, is untested. The concern is specific: blocking artifacts follow a grid that spans all channels, and depthwise filters lack cross-channel spatial capacity.
Research Program
Data. We re-encode Aff-Wild2 face tracks with the HEVC reference encoder HM16.20, the setting used by CPGA and MF-DLUT, at QP 22, 27, 32, 37 and 42. We also re-encode with an H.264 encoder. In both cases we log per-block QP, frame type, residual energy and motion vectors. Aff-Wild2 sources are already YouTube-compressed, so this is double compression. We treat the original encode as part of the "unconstrained" condition and report it separately.
Model. The frame encoder is a MobileNet-style depthwise separable network over face crops. Codec metadata is resampled onto the crop as a small reliability map. It enters the model in two places. First, it scales intermediate features region by region through a lightweight conditioning layer. Second, it is a frame-level input to an Affective-Processes-style context model, where it biases which frames serve as context anchors. The motivating intuition is that a low-QP I-frame should anchor the context and a high-QP B-frame should be bridged.
Diagnostic study. Before any modelling, we test the premise. We measure whether per-frame errors of a strong uncompressed-trained model track QP and position within the group of pictures (the repeating I/P/B frame pattern). If errors are flat across those positions, the core claim is falsified early.
Factorization study. We train a standard ResNet-18 affect model compression-aware. We then (a) factorize it after training with GDWS and (b) train a native depthwise separable model of matched FLOPs. For both, we measure how the accuracy-versus-QP curve changes. This tests whether the "expressiveness" worry about depthwise filters on dense degradations holds in practice.
Baselines
The naive reference is a standard frame-wise affect model trained on the original Aff-Wild2 and tested on re-encoded video with no compression awareness. It measures how much damage there is to recover. For context on clean data, the 8th ABAW competition reports a winning valence-arousal CCC of 0.600 against a baseline of 0.201. Its top expression F1 is 0.365 against a baseline of 0.205. ABAW2 reports a VGG-FACE baseline mean CCC of 0.22. We found no reported result on Lacuna for any method on Aff-Wild2 re-encoded at controlled QPs, nor for Affective Processes on Aff-Wild2. Our compressed-condition numbers will therefore be new reference points, and the clean-data numbers are only a ceiling.
The strongest competitor is our own temporal model with the same depthwise separable backbone and the same Affective-Processes-style context module, trained with compression augmentation across the full QP range and both codecs, but given no codec metadata. This is the baseline most likely to close the gap. It sees the same data and the same damage, and a large network may learn to infer quality internally. It gets an identical training budget and an identical hyperparameter search. The proposal beats it only if it improves valence-arousal CCC and action-unit F1 at QP 37 and 42 by margins that exceed seed-to-seed variance over at least three seeds, with smaller or no gains at QP 22.
The decoupled pipeline runs CPGA or RivuletMLP as a pre-filter in front of the same augmented affect model. It is the natural "use existing tools" alternative. It is reported with combined accuracy and end-to-end throughput.
The compressed-domain baseline is a reimplementation of the IFERCV residual branch, for expression recognition only, matched in parameters.
The key-ingredient ablation replaces the codec reliability map with a degradation estimate predicted from decoded pixels, in the manner of Sensitivity Decouple Learning, trained to regress QP. This isolates whether the bitstream itself matters or merely the act of quality-aware weighting.
Why not just…?
Why not just train with compression augmentation? This is the real threat, and it is the strongest baseline above rather than a straw man. Nothing in the file shows that augmentation fails. The argument for codec metadata is only that it is exact and spatially resolved, whereas inferred quality may be confounded with genuine low-texture or low-light faces. If the augmented model matches ours at high QP, we will report that the metadata is unnecessary.
Why not just restore the video first with CPGA or RivuletMLP? These models minimize pixel error, and the file gives evidence that MSE-oriented processing smooths away expression cues. Optimized Learned Image Compression for FER reports a +4.04% BD-accuracy gain from task-aware rather than distortion-only optimization, though for images and at the encoder side. Restoration is also slow: RivuletMLP reports 5.2 FPS for CPGA and 23.2 FPS for its light model at 720p, before any recognition runs. The decoupled-pipeline baseline measures directly whether restoration helps affect accuracy, regardless of these priors.
Why not just let the model's own uncertainty pick context frames, as Affective Processes does? That may work, and it is ablated by removing the codec input to context selection. Our hypothesis is that compression produces confidently wrong frames, for example blocking that looks like a frown. If so, self-uncertainty will be miscalibrated exactly where codec QP is high. We test this by comparing the model's uncertainty against QP-stratified error.
Proposed Experiments and Decisive Checks
The main table reports valence-arousal CCC, 12-AU average F1 and expression macro-F1 on the Aff-Wild2 validation split at each QP and codec. It also reports a jitter score: the mean absolute frame-to-frame change in predictions minus that of the ground truth, which captures flicker-induced instability. Efficiency is reported as FLOPs and measured FPS on a Jetson Xavier, matching the GDWS hardware so throughput claims are comparable. Generalization is checked by training on HEVC and testing on H.264 metadata.
Three results decide the proposal. The diagnostic must show errors that are periodic with GOP position and that grow with QP. The codec-conditioned model must beat both the augmented baseline and the pixel-estimated-reliability ablation at QP 37 and above. The depthwise separable model must stay within about 1 CCC point of the dense model at every QP, the tolerance GDWS observed for adversarial robustness. If the gap instead widens with QP, the file's warning about depthwise expressiveness under dense degradations is confirmed. That would be a useful negative result for edge deployment.
References
- Identity-aware Facial Expression Recognition in Compressed Video
- Optimized Learned Image Compression for Facial Expression Recognition
- Affective Processes: Stochastic Modelling of Temporal Context for Emotion and Facial Expression Recognition
- CPGA: Coding Priors-Guided Aggregation Network for Compressed Video Quality Enhancement
- Multi-Frame Deformable Look-Up Table for Compressed Video Quality Enhancement
- RivuletMLP: An MLP-based Architecture for Efficient Compressed Video Quality Enhancement
- Sensitivity Decouple Learning for Image Compression Artifacts Reduction
- Xception: Deep Learning with Depthwise Separable Convolutions
- Generalized Depthwise-Separable Convolutions for Adversarially Robust and Efficient Neural Networks
- Advancements in Affective and Behavior Analysis: The 8th ABAW Workshop and Competition
- Analysing Affective Behavior in the second ABAW2 Competition