Masked Pre-Training of a Small Hybrid Face Encoder for End-to-End Concordance Training

Systems that track a person's emotion frame by frame in video are usually trained in two separate pieces. A large image model is frozen and turns each frame into a feature vector, and a small temporal model is then trained on those cached vectors. The training objective is the concordance correlation coefficient (CCC), a score that rewards predictions which rise and fall with the human rating curve at the right level and amplitude. CCC is computed from statistics of a whole window of frames, so it gives reliable gradients only when windows are long. Long windows are exactly what make it too expensive to send gradients back into a large frozen image model. The result is that the part of the system that sees the face never learns from the objective that matters.

We propose to remove that bottleneck. The plan is to build a face encoder of only a few million parameters that mixes convolutions and attention, and to pre-train it without labels by hiding facial regions in video and asking it to reconstruct them. That gives it strong features at a cost low enough to train the whole pipeline, pixels to predictions, with CCC over windows of hundreds of frames. The core claim is that, for continuous affect tracking, a small encoder trained end to end on long windows beats a large encoder frozen and used through short windows. The claim also requires that masked pre-training is what makes the small encoder good enough to start from.

Core Claim and Why It Is New

CCC between a predicted sequence $x$ and a rating sequence $y$ is $2\rho\sigma_x\sigma_y / (\sigma_x^2+\sigma_y^2+(\mu_x-\mu_y)^2)$. Here $\rho$ is their Pearson correlation, $\mu$ are the means and $\sigma^2$ the variances, all estimated inside one training window. The usual sampling result for a correlation estimate is that its variance scales roughly as $(1-\rho^2)^2/T_{\text{eff}}$. Here $T_{\text{eff}}$ is the effective number of independent frames in the window. Affect ratings change slowly and are strongly autocorrelated, so $T_{\text{eff}}$ is much smaller than the frame count, and short windows give noisy gradients. Work using CCC as a loss treats it as better than squared error because it keeps prediction amplitude rather than flattening toward the mean (Weninger et al.; Jin et al.). The research file itself flags small-batch and short-window instability as an open problem.

Current systems avoid the issue by decoupling. Min et al. freeze an ImageNet-21k ViT-Base, cache frame features, and train a temporal Transformer with random frame masking and CCC loss. VEATIC runs a ResNet-50 per frame into a ViT over 5-frame windows, trained with mixed squared error and CCC. Cabacas-Maso et al. put an LSTM on a MobileFaceNet-based convolutional network. None of these trains the spatial encoder through a long-window CCC loss.

The closest pre-training work is MARLIN. It pre-trains a pure ViT on unlabeled face video by masking facial regions (eyes, nose, mouth) through time, and evaluates mostly on classification-style tasks. It also inherits ViT cost, which is exactly what keeps end-to-end long-window training out of reach. What we add is a different encoder family, a fix that makes masked pre-training work on that family, and a test of the specific hypothesis that end-to-end CCC on long windows is the payoff.

Research Program

The encoder borrows its spatial design from Lite-Mono. Lite-Mono interleaves stacked dilated convolutions, which widen the local view, with "local-global interaction" blocks. Those blocks attend across channels rather than across positions, so cost grows linearly with image size. Lite-Mono reaches 3.1M parameters and 20 FPS on a Jetson Xavier, against 2 FPS for a tiny ViT variant. That depth-estimation result motivates the cost argument here, but it does not show the design works for faces. We add a lightweight temporal Transformer over per-frame embeddings, so the full model processes a 256-frame clip of 112-pixel face crops on one accelerator with gradients enabled.

Masked pre-training of this hybrid is not off-the-shelf. Inserting masked or blank patches breaks convolutions. A²MIM solved this for plain CNNs in three ways. It masks intermediate feature maps after the early layers. It fills masked locations with the image's mean color instead of a learned token. It reconstructs with a frequency-weighted loss that emphasizes edges and contours. A²MIM is the source of the method we reuse.

The hybrid adds a problem A²MIM never faced. Channel-wise attention builds a channel-by-channel covariance from all spatial positions, so mean-filled positions drag that covariance toward the fill statistics. We therefore compute the covariance only over visible positions, renormalized by the visible count. We call this masked cross-covariance attention, and we treat it as a first-class ablation.

Masks follow MARLIN's facial-region tube masking. A face parser picks regions, and the same regions stay hidden across a short clip so the model cannot copy from neighboring frames. That design follows VideoMAE, which showed temporal redundancy otherwise makes reconstruction trivial. Pre-training uses YouTube Faces as in MARLIN, via the MARLIN code for parsing and the A²MIM code for intermediate masking and the frequency loss.

Fine-tuning trains everything end to end on Aff-Wild2 valence and arousal. The loss is window-level CCC, with a small squared-error term for stability as VEATIC found necessary. Window length $T$ is varied from 8 to 256 frames. Unannotated or dropped frames are excluded from the window statistics rather than interpolated.

We deliberately do not put CCC into the reconstruction loss. Per-patch normalized-pixel targets already remove patch mean and scale. Squared error on such targets differs from a per-patch CCC loss mainly through the variance-matching term, so we expect little from that combination.

Baselines

The naive reference points come from ABAW8. Its official baseline reports a combined valence-arousal CCC of 0.201 on the Aff-Wild2 validation set. A second naive reference is our own hybrid trained from random initialization with the same end-to-end CCC recipe, which tells us whether pre-training is carrying the result.

The strongest published system is the ABAW8 winning team, reported at 0.600 validation valence-arousal CCC in the ABAW8 paper. We cannot rerun it under matched conditions, so it serves as a ceiling to approach, not a controlled comparison. Two published systems we can match more closely are the following. DDAMFN with LSTM reports 0.480 combined CCC (Cabacas-Maso et al.). This is the strongest small convolutional encoder trained with CCC, and it is the natural "no masked pre-training" competitor. Frozen ViT-Base with temporal frame masking reports 0.32 average CCC (Min et al.).

The most dangerous controlled baseline is frozen MARLIN features with a temporal Transformer trained by CCC on long windows. Freezing makes long windows cheap there, so this isolates whether end-to-end spatial adaptation matters at all. We found no reported MARLIN result on Aff-Wild2 valence-arousal.

All controlled baselines use the same face crops, the same official training and validation split, and the same temporal head. Each gets the same fine-tuning GPU-hour budget and the same number of hyperparameter trials over learning rate, window length and loss mix. The ablations isolate the key ingredients. They are frozen versus end-to-end for our encoder at each $T$; masked versus unmasked covariance in the attention blocks; facial-region versus random tube masks; and A²MIM-style pre-training versus ImageNet-supervised initialization of the same hybrid. Beating the field means two things on the validation set. First, exceeding DDAMFN+LSTM's 0.480 and the frozen-MARLIN baseline under matched budgets. Second, showing a monotonic gain with $T$ that appears only when the encoder is trained.

Why not just…?

Why not just freeze MARLIN and train a long-window temporal model? This is the real threat. MARLIN's region masking already teaches facial dynamics, and freezing lets the temporal model see long windows cheaply. If frozen MARLIN at $T=256$ matches our end-to-end hybrid, the core claim is wrong and we will say so. The frozen-versus-end-to-end sweep over $T$ for both encoders settles it directly.

Why not fine-tune a pure ViT end to end with gradient checkpointing or low-rank adapters? Memory tricks can make long windows feasible for ViT-Base at a large compute cost. We include a compute-matched ViT run in which adapters are fine-tuned through CCC at the longest $T$ the budget allows. If it wins at equal GPU-hours, the hybrid is only an efficiency contribution, not an accuracy one.

Why not train a small pretrained CNN end to end with CCC and skip masked pre-training? DDAMFN+LSTM already reaches 0.480, so supervised face-recognition initialization may be enough. The A²MIM evidence that masked pre-training beat supervised ResNet-50 (80.4% versus 79.8% on ImageNet) concerns static classification. It is suggestive, not decisive, for affect. The ImageNet-initialized and scratch-initialized hybrid ablations answer this question, and so does the comparison at 10% of Aff-Wild2 labels, where MARLIN reported its largest advantages.

Why not just add Min et al.'s frame masking to a frozen encoder? That trick regularizes the temporal model but cannot change what the encoder sees. We apply it to every temporal head, including ours, so any remaining gap comes from encoder adaptation.

Proposed Experiments and Decisive Checks

The primary benchmark is Aff-Wild2 valence and arousal, reporting valence, arousal and mean CCC on the official validation split (Aff-Wild2). The secondary benchmark is transfer to VEATIC, where the published ResNet-50+ViT model reports valence CCC 0.6678 and arousal 0.6296 with face plus context. That tests whether the encoder helps when scene context also matters.

The decisive check is the interaction between window length and training regime. The proposal predicts that end-to-end CCC improves steadily as $T$ grows to 256 while frozen encoders flatten early, and that this interaction disappears without masked pre-training. A second check measures gradient noise directly, as the variance of the CCC gradient across resampled windows. That variance should fall with $T$ in roughly the way the correlation-estimator argument predicts, which would tie the accuracy gains to the mechanism rather than to extra capacity. A third check shows whether masked covariance attention lowers reconstruction loss and raises downstream CCC relative to the unmasked version; if it does not, the A²MIM recipe transfers to hybrids without change, which is itself worth reporting. Throughput on a single accelerator is reported for every configuration, so the efficiency claim can be checked against the accuracy one.