When a person presses their lips together, squints, or begins to smile, the line where the upper and lower lip meet, or where the two eyelids close, often has almost no visible edge. Two regions of similar skin simply touch. Video face parsing models label every pixel with a facial part such as upper lip, lower lip, or eye. They handle these moments worst, and they do it frame by frame: the contact line wanders and flickers, and it vanishes or reappears as the mouth opens and closes. We propose to treat touching facial parts the way cell-segmentation work treats touching nuclei, as separate instances whose internal geometry reveals where they meet. We then carry that geometry forward in time along the face's own motion field, so that when image contrast disappears the contact line is still pinned by where the tissue came from.
The opportunity is concrete. The first- and third-place systems in the CVPR 2021 Short-video Face Parsing challenge both process frames essentially independently. Both conclude that temporal consistency is unaddressed future work (Shuffle Transformer with Feature Alignment for Video Face Parsing; 3rd Place Solution for Short-video Face Parsing Challenge). Dedicated facial motion models now separate head rotation from expression-driven skin motion (FacialFlowNet), but they are known to smear motion across the very lip and nostril borders that parsing struggles with (Facial optical flow estimation via neural non-rigid registration). Each field has the piece the other lacks.
Core Claim
The claim is that the right quantity to track through a facial expression is a per-part offset field rather than a label map or a boundary map. An offset field is a two-channel image in which every pixel points to the nearest point on the skeleton of its own facial part. Where two parts touch, neighbouring offsets point in roughly opposite directions. The contact line therefore shows up as a sharp flip in vector direction even when intensity carries no edge. This is the mechanism that Accurate Nuclear Segmentation with Center Vector Encoding used to separate touching nuclei without guessing edges. It reached an Aggregated Jaccard Index of 0.561 on the Kumar pathology set, against 0.508 for the prior method.
Lips and eyelids are thin and curved, so a single centroid would fall outside the part. We therefore point to the nearest skeleton point instead. The flip survives this change: pixels on either side of a lip contact line point to different skeletons in opposite directions.
We define a contact score at pixel $x$ as
$$c(x) = \max_{y \in N(x)} \tfrac{1}{2}\left(1 - \cos\angle\big(o(x), o(y)\big)\right),$$
where $o(x)$ is the predicted offset at $x$ and $N(x)$ is its 4-neighbourhood. The score is near 1 on contact lines and near 0 inside parts.
The temporal half of the claim is that this field should be propagated along expression motion. The motion should itself be prevented from smoothing across high-$c$ pixels. Flow then carries the contact line through low-contrast frames, and the contact line in turn stops the flow from bleeding across it. We expect the gains to concentrate in two places. The first is frames with an active contact line: closed mouth, pressed lips, squint. The second is the fast onset phase of an expression, where motion blur and non-uniform skin stretching are largest. Temporal expression dynamics thus serve as the evaluation lens, not as a recognition target.
Research Program
The per-frame model is a standard high-resolution parsing network, an HRNet-W48 with an Object-Contextual Representation head as in the EANet system. It gains an extra head that predicts the skeleton-offset field for each part group. The groups are lips, eyes and brows, and the nose–mouth region. Offset targets are computed automatically from the existing part masks, so no new annotation is needed for training. At inference, part labels come from the parsing head everywhere except at high-$c$ pixels. There, each pixel is assigned to the part whose skeleton its offset points toward, much as predicted centroids and membranes act as seeds and barriers in Eschweiler et al..
For time, we take total facial flow from DecFlow (FacialFlowNet), which is trained on 105,970 synthetic pairs with ground-truth flow. We use it to warp the previous frame's skeleton positions, and we recompute a propagated offset field from the warped skeletons. Offsets cannot simply be warped as images, because the target point moves too. The current and propagated fields are fused with a learned per-pixel confidence. That confidence is driven by local image contrast: where contrast is low, the propagated field wins. The flow is masked with $1 - c$ during its smoothness refinement, so it may be discontinuous across contact lines.
Only a fraction of Short-video Face Parsing frames are annotated, roughly 25,000 across about 1,500 videos. We therefore train with a supervised loss on labeled frames plus a temporal term on unlabeled neighbours:
$$L = L_{\text{parse}} + \lambda_o L_{\text{offset}} + \lambda_t \sum_x \big(1 - c(x)\big)\,\lVert o_t(x) - \tilde o_{t}(x)\rVert_1,$$
where $\tilde o_t$ is the offset field propagated from frame $t-1$. The weight $1 - c(x)$ relaxes the temporal penalty exactly at contact lines, where topology may legitimately change as a mouth opens.
Topological events need explicit handling. When the mouth opens, an inner-mouth region appears between the lips and the contact line should dissolve rather than be propagated. The confidence gate must learn to hand control back to the image when the current frame shows strong new evidence. We will measure whether it does.
Baselines
The naive reference is the per-frame HRNet-W48 + OCR parser, trained identically but without the offset head or temporal term. On Short-video Face Parsing, Shuffle Transformer with Feature Alignment for Video Face Parsing reports a validation mIoU of 0.7283 for HRNet-W48. That gives a sanity check that our reimplementation is in range.
The strongest existing method is the challenge winner itself. It is a Shuffle Transformer backbone with Feature Alignment Aggregation, a module that learns offsets to realign upsampled coarse features to high-resolution edges. It reports 86.9519% challenge accuracy and 0.7312 validation mIoU, or 0.7338 when ensembled with HRNet-W48. It is the strongest competitor because it already attacks jagged, misaligned part boundaries under non-rigid motion, which is our target failure. We add our offset head and temporal term on top of the same backbone. This separates our contribution from backbone choice, and it lets the comparison inherit the tuning the winners already did. The edge-aware alternative is EANet, which uses an edge head with an edge attention loss restricted to boundary pixels. It reports 84.94% alone and 86.16% with ensembling and Grabcut (3rd Place Solution). We rerun it single-model without Grabcut so that post-processing does not confound the comparison.
We found no reported result on Lacuna for temporal flicker or contact-line accuracy on this benchmark. Those numbers will be produced by us for every system under one protocol. All systems get the same training frames, the same unlabeled neighbours where the method can use them, the same number of GPU-hours, and the same hyperparameter search budget.
Beating the winner means matching its overall mIoU within noise, which should be expected since most pixels are interior. It also means reducing contact-line error and flicker by a margin larger than the seed-to-seed spread.
Two ablations isolate the key ingredient. The first keeps the offset head but drops temporal propagation, which tests the instance representation alone. The second keeps temporal propagation but propagates the label map instead of the offset field. That second ablation is the decisive one.
Why not just…?
Why not just run the best per-frame parser and median-filter its outputs over time? Filtering suppresses flicker but cannot recover a boundary that no frame located correctly. It also lags during fast onsets, which is exactly when the lips move most. In continuous emotion tracking, Allognon et al. applied post-hoc median filtering and still reached only 0.264 arousal CCC, a sign that smoothing cannot restore signal missing at the frame level. The EANet authors likewise note that their frame-level Grabcut refinement does not produce temporal consistency. We include a tuned temporal median filter on the per-frame winner as a baseline. If it matches us on contact-line error during onset frames, the temporal half of our proposal is unnecessary.
Why not just warp the previous frame's parsing mask with facial flow and fuse it with the current prediction? This is the real threat, since it is the standard video-propagation recipe and DecFlow flow is good. Our reason to expect it to fall short is specific. Facial flow smooths across lip and nostril borders (Peng et al.), so a warped label map inherits that bleeding precisely at contact lines. The offset representation both marks the discontinuity and masks the flow there. The label-propagation ablation settles the question directly. If it matches the full model on contact-line error, the novelty collapses to "use better flow," and we would report that.
Why not just add a stronger boundary loss or boundary branch? Boundary-weighted losses do help on static faces. BSPNet reports 2–3% gains on small parts like eyes and brows (A High-Efficiency Framework for Constructing Large-Scale Face Parsing Benchmark). But an edge head needs an edge in the image, and a pressed-lip contact line often has none. The offset flip is defined by which side a pixel belongs to, not by local contrast, which is why it worked for touching nuclei. EANet and the per-frame offset ablation, compared on a low-contrast subset of contact-line pixels, test this claim.
Why not just add a ConvLSTM on the backbone features? Generic recurrence on facial feature maps has given marginal or negative gains for expression trajectories (Güçlütürk et al.; Hassani). Recurrent instance models tend to produce blobby masks that need CRF cleanup (Recurrent Instance Segmentation). We still include a ConvLSTM variant at equal parameter count, so that the benefit is attributed to explicit geometric propagation rather than to added capacity.
Proposed Experiments
The primary benchmark is Short-video Face Parsing, reporting mIoU and challenge accuracy. We add three new measurements. The first is contact-line error: the mean distance between predicted and ground-truth contact polylines, where ground truth is extracted wherever two part labels in the same group meet directly. We will verify which lip and eye categories the label set distinguishes. We will also hand-annotate contact polylines on a few hundred held-out frames to check the automatic extraction. The second is flicker: disagreement between consecutive predictions after warping with flow, computed on unlabeled frames and restricted to part boundaries. The third is topology accuracy: the fraction of frames where the model gets the open-versus-closed state of the mouth and eyes right.
For the stratified analysis, frames are binned by expression phase. We estimate phase from the magnitude of DecFlow's expression flow, labeling rising magnitude as onset, peak as apex, and falling magnitude as decay, in the sense of Bartlett et al.. We predict that the full model's advantage over the winner is largest on onset frames with an active contact line and near zero on interior pixels of neutral frames. A uniform improvement would suggest a generic regularization effect rather than the claimed mechanism.
Three further checks probe generality. On LaPa and HELEN we train only the per-frame offset variant, to confirm that the representation does not hurt static parsing. The BSPNet HELEN F1 of 91.0% serves as the reference point. On the RGB-D facial set and FaceFlowData from Peng et al., we test whether contact-masked flow lowers end-point error near the lips. The reference is their reported average end-point error of 7.32, against 11.29 for RAFT. We also report failures on mouth-opening events honestly: the method is supposed to let a contact line dissolve, and if propagation keeps it alive, the confidence gate has failed.
The decisive result is the pairing of two comparisons: offset-field propagation beating label-map propagation on contact-line error in onset frames, while the per-frame offset head alone does not close that gap.
References
- Shuffle Transformer with Feature Alignment for Video Face Parsing
- 3rd Place Solution for Short-video Face Parsing Challenge
- FacialFlowNet: Advancing Facial Optical Flow Estimation with a Diverse Dataset and a Decomposed Model
- Facial optical flow estimation via neural non-rigid registration
- Accurate Nuclear Segmentation with Center Vector Encoding
- CNN-based Preprocessing to Optimize Watershed-based Cell Segmentation in 3D Confocal Microscopy Images
- Continuous Emotion Recognition via Deep Convolutional Autoencoder and Support Vector Regressor
- A High-Efficiency Framework for Constructing Large-Scale Face Parsing Benchmark
- Deep Impression: Audiovisual Deep Residual Networks for Multimodal Apparent Personality Trait Recognition
- Facial Affect Estimation in the Wild Using Deep Residual and Convolutional Networks
- Recurrent Instance Segmentation
- A Prototype for Automatic Recognition of Spontaneous Facial Actions