A Learned Shape–Muscle Prior for Semi-Supervised Facial Action Unit Intensity

Facial action units (AUs) are the coded muscle movements of the Facial Action Coding System, such as "cheek raiser" or "lip corner puller." Estimating their intensity from video is limited mainly by labels. Only certified coders can annotate AUs, so datasets are small and dominated by neutral frames, while unlabeled face video is effectively free. Faces are also strongly structured. When AU12 (the smile muscle) fires, the mouth corners move outward and up. When AU1 and AU2 co-occur, the brows rise together. This proposal builds a model that learns that structure once, from labeled data, as a prior over what a valid joint output looks like. The prior then supervises the network on unlabeled frames.

The approach treats a network's output as a stack of images. One image per facial landmark marks where that point is. One image per AU places a Gaussian blob over the muscle, with the blob's height encoding intensity; this output format is called heatmap regression. We train a small denoising network on the ground-truth stacks only, so it learns to map a corrupted or implausible stack back to a plausible one. On unlabeled frames, the main predictor is penalized by how far its output sits from the prior's correction. The claim is that a prior over the coupled landmark and AU heatmaps lets unlabeled video improve AU intensity estimation, and that the gain is largest where predictions are uncertain: occlusion and large head pose.

Core Claim

Existing work has each piece separately. Joint Action Unit localisation and intensity estimation through heatmap regression encodes AU intensity as Gaussian amplitude in a single Hourglass network and reaches 0.68 average intra-class correlation (ICC, an agreement score) on BP4D. It models no landmarks and imposes no structure beyond the network's receptive field. A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation gets landmark knowledge by adapting a pretrained face-alignment network, reaching 0.64 ICC on FERA 2017. The transfer is sequential, and it uses no unlabeled data.

Implicit constraint learning in faces has so far lived only in vector space. Weakly Supervised Facial Action Unit Recognition Through Adversarial Training matches 1D AU vectors to a hand-written rulebook. Dual Semi-Supervised Learning for Facial Action Unit Recognition closes a loop between AU vectors and landmark coordinate vectors and reaches 0.8287 F1 on CK+ with 10% labels. Both motivate our proposal, because they show that output-space structure substitutes for labels. Neither touches dense spatial outputs or continuous intensity on spontaneous data.

In body pose, Adversarial PoseNet supplies the idea of judging plausibility directly on heatmaps. It also constrains our design: it needs two discriminators and is unstable to train. Adversarial Constraint Learning for Structured Prediction warns that unanchored output priors drift toward trivial solutions. We therefore use a non-adversarial denoising prior, and we always train it alongside a supervised anchor.

The new element is a learned prior on the joint landmark-plus-AU heatmap tensor, used as an unlabeled-data loss. Our hypothesis is that the heatmap form carries more information than vectors, because heatmaps keep the spread and multiple peaks that signal uncertainty. A prior over a diffuse mouth-corner heatmap and a hesitant AU12 blob can resolve both together. A prior over a single averaged coordinate cannot, since the averaging has already happened. This hypothesis may be wrong, and the experiments are designed to show it if it is.

Research Program

The predictor $f$ is a stacked Hourglass with two output groups that share one backbone. The first group gives 68-point landmark heatmaps. The second gives variable-amplitude AU heatmaps, using the target construction of the BP4D heatmap paper above. Supervised frames get the usual pixelwise loss on both groups.

The prior $P$ is a small fully convolutional encoder-decoder trained only on ground-truth stacks $Y$. Its inputs are corrupted by a function $c$ that mimics real predictor failures: blurring peaks, splitting peaks into two modes, jittering subsets of landmarks, and swapping or scaling AU amplitudes in ways that break co-occurrence. It is trained to undo them:

$$\min_P \; \mathbb{E}_Y \, \lVert P(c(Y)) - Y \rVert^2 ,\qquad L_u(x) = \lVert f(x) - \mathrm{sg}[P(f(x))] \rVert^2 ,$$

Here $L_u$ is the loss applied to an unlabeled frame $x$, and $\mathrm{sg}$ stops gradients so that only the predictor moves. The total objective adds $\lambda L_u$, with weight $\lambda$, to the supervised losses. $P$ is frozen during this stage, which keeps the training a plain regression problem rather than a two-player game.

Three design questions are analyzed directly. First, does coupling matter? We compare one joint prior against separate landmark-only and AU-only priors. Second, does the heatmap form matter? We compare against an identical prior acting on landmark coordinates and scalar intensities read from the heatmaps. Third, should the corruption family come from real errors? We compare hand-designed corruptions against residuals of a supervised-only model.

Baselines

The naive reference point is the supervised-only joint Hourglass trained on the labeled subset. It uses identical architecture, data split, and training schedule, with $\lambda = 0$. Every gain is measured against it, at every label fraction.

The strongest published heatmap method on the intensity task is the single-task Hourglass of Joint Action Unit localisation and intensity estimation through heatmap regression. It reports 0.68 average ICC and 0.94 MSE on BP4D over its five AUs. We reproduce it on our splits at 100% labels, so the comparison uses the same AU set and the same tuning budget.

On FERA 2017 the reference is the face-alignment transfer model of A Transfer Learning Approach to Heatmap Regression for Action Unit Intensity Estimation, at 0.64 ICC. It is the strongest competitor for the specific claim that landmark knowledge helps AUs, because it obtains that knowledge by transfer rather than by a prior. We run its adaptation-layer variant from the same pretrained weights, with the same labeled subsets.

On DISFA the relational reference is Dynamic Probabilistic Graph Convolution for Facial Action Unit Intensity Estimation. It reports 0.56 average ICC, against a previous best of 0.50. Its graph explicitly models AU correlation on pooled features, so it tests whether a dense output prior adds anything beyond explicit relational modeling.

For semi-supervised comparison we implement a vector-space dual-learning prior in the spirit of Dual Semi-Supervised Learning for Facial Action Unit Recognition on the same backbone and unlabeled frames. No intensity numbers from that paper are found in the file; only its CK+ detection F1 is reported. We also run an Adversarial PoseNet-style heatmap discriminator in place of the denoising prior, which isolates the choice of prior type.

Beating the field means two things. First, ICC should rise above the supervised-only model and the transfer baseline at 10% and 25% labels on BP4D and DISFA, by more than the seed-to-seed spread over three runs. Second, at 100% labels the method should at least match the published single-task numbers.

Why not just…?

Why not just train the shared two-head Hourglass with no prior? The receptive-field argument in Joint Action Unit localisation and intensity estimation through heatmap regression says that correlation is already captured implicitly, and with full labels that may be true. The plain multi-task model is our naive baseline. The proposal's claim lives in the low-label regime, where a model without the prior has nothing to learn from on unlabeled frames. If the plain joint model matches the prior at 10% labels, the claim fails.

Why not just put the prior on landmark coordinates and AU scalars? This is the most serious threat. Vector priors are cheaper, and much of the shape–muscle coupling, such as mouth width rising with AU12, is visible in coordinates alone. Our matched vector-prior ablation settles it. We predict near-parity on clean frontal frames and a heatmap advantage only on occluded and non-frontal frames. If the advantage does not appear there, we will report that the vector prior suffices and that heatmaps contribute only through the predictor.

Why not just fix the multi-task optimization with gradient balancing? RotoGrad rotates and rescales task gradients to reduce interference. That could explain any gain we see from joint training, through landmark gradients no longer swamping the AU gradients. We add RotoGrad to the supervised-only baseline. The prior must improve on that stronger baseline, not just on static loss weights. Balancing gradients does not create a training signal from unlabeled frames, so the two should stack if the claim holds.

Why not just transfer from a landmark network, as is done today? Transfer is the strongest practical baseline, and it is run directly. The prior differs in two ways. It is trained on AU-annotated stacks, so it encodes muscle co-activation, which a landmark network cannot know. It also uses target-domain unlabeled video. A combination of transfer initialization and the prior is also reported.

Proposed Experiments

All experiments use subject-independent folds on BP4D (the five-AU intensity set) and DISFA (twelve AUs), with FERA 2017 multi-view for pose. Labeled fractions are 10%, 25%, and 100%, with the remaining frames used unlabeled. BP4D ships with landmark annotations. Where landmark labels are missing, landmark heatmaps are supervised only on frames that have them.

Metrics are ICC and MSE per AU and averaged, plus normalized mean landmark error. We also report a structure score: the divergence between predicted and ground-truth pairwise AU co-activation matrices on the test set.

There are three decisive checks. The first is the low-label curve against the supervised-only and transfer baselines. The second is the heatmap-versus-vector prior gap, measured separately on clean frames, on frames with synthetic occlusion masks over the eyes or mouth, and on non-frontal FERA 2017 views. The third is cross-dataset transfer from BP4D to DISFA, where the prior is trained on BP4D stacks but applied to unlabeled DISFA frames. This tests whether the learned shape–muscle structure carries across subjects and recording setups.

A positive result requires a clear low-label gain and a heatmap-over-vector gap concentrated under occlusion and pose. A gain with no such gap would mean the useful ingredient is the prior itself and not its spatial form. We would report that as the finding.