Unlearning Symmetry: Hemiface-Gated Attention for Few-Shot Landmark Detection on Asymmetric Faces

Facial landmark detectors, the models that place points on the eyes, brows, nose, mouth and jaw, are increasingly used as measuring instruments in medicine. Clinicians use them to quantify how far a paralyzed mouth corner lifts, or how completely a paretic eye closes. These detectors learn from large collections of mostly healthy, roughly symmetric faces. On a face with one-sided paralysis they tend to "pull" the points on the affected side back toward where a symmetric face would put them. That error falls on exactly the asymmetry the clinician is trying to measure. The newest accurate detectors are hybrid CNN-Transformer models: a convolutional network extracts local edges and textures, and a Transformer lets every part of the face attend to every other part. We argue that this global attention is a likely carrier of symmetry bias, because it lets the healthy half of the face vote on where the paralyzed half's landmarks should be. We also argue that the bias cannot be fine-tuned away when only a few dozen patients are available.

We propose to measure this "symmetry pull" directly and then remove it with a very small intervention. We take a hybrid detector pretrained on large benchmarks and freeze it. We then learn, from 10 to 100 patients, a handful of scalar gates that control how much attention may flow across the facial midline. If the claim holds, a clinic could adapt a large pretrained hybrid detector to a rare asymmetric condition by changing a few dozen numbers. The adapted model would keep its benchmark-level precision on the healthy side and stop hiding the pathology on the affected side.

Core Claim

Our claim has two parts, and both can be tested. The first is diagnostic. In hybrid landmark detectors, the share of attention flowing between left and right hemifaces predicts how much the model under-reports a real one-sided deformation. We expect this under-reporting to be larger than in purely convolutional heatmap detectors with comparable benchmark error, because in a CNN the influence of the opposite hemiface must pass through many local layers.

The second part is interventional. Suppressing cross-midline attention with learned per-head gates, fitted on as few as 10–50 patients while the backbone stays frozen, should reduce asymmetry under-reporting more than three alternatives given the same data: full fine-tuning, parameter-efficient fine-tuning, and regularized transfer. It should do so without degrading accuracy on healthy faces.

The primary outcome is a clinically meaningful quantity, not overall landmark error. We call it signed asymmetry error: the predicted left-right difference in a clinical measure (for example oral commissure excursion or eye opening) minus the true difference. Symmetry bias shows up as a systematic negative value. Average error over all points can hide it, because the healthy side dominates.

Grounding in Prior Work

Three findings motivate the proposal. Toward an Automatic System for Computer-Aided Assessment in Facial Palsy retrained a cascade of regression trees on the MEEI facial nerve repository and named the symmetry bias. Its model trained on 11,500 healthy 300-W images reached a normalized root-mean-square error of 8.56 on palsy patients, against 6.03 after retraining on 1,440 disease-specific images. Atypical Facial Landmark Localisation with Stacked Hourglass Networks found that convolutional hourglass models beat statistical shape models on palsy faces, with the worst failures at the mouth. Limitations and Biases in Facial Landmark Detection showed that fine-tuning narrows but does not close the gap between patient groups. All three studied shallow or purely convolutional models, so none says whether global attention helps or hurts.

The method comes from the hybrid landmark literature. Cascaded Dual Vision Transformer (D-ViT) pairs a ResNet with cascaded blocks. Each block contains a spatial Transformer, whose tokens are image patches, and a channel Transformer, whose tokens are feature channels that combine into landmark heatmaps. Long skip connections carry low-level features forward through the cascade. D-ViT reports a normalized mean error (NME) of 3.75 on WFLW, 4.13 on COFW and 2.85 on 300W. Its channel attention is designed to learn fixed geometric relations between landmarks, which is precisely what should break on a face whose two halves no longer move together. We reuse D-ViT as the frozen backbone and intervene inside its attention.

ORFormer (WFLW NME 3.86) adds "messenger tokens" that estimate each patch from all other patches and replace patches that disagree with the consensus. This design constrains ours: on a palsy face, an eye that fails to close disagrees with the healthy consensus and could be "repaired" into a healthy eye. We treat ORFormer as a second test subject for the diagnostic, not as a method to extend.

Two convolutional small-sample methods supply our strongest competitors. Few-Shot Model Adaptation freezes an autoencoder pretrained on 2.1M faces and adds small convolutional transfer layers. It reports 4.71% NME with 5% of 300W, against 3.88% with the full set. Regularized Transfer Learning for FAS fine-tunes a ResNet-50 while forcing its spatial attention maps to match a pretrained model (RTL-SAM). On 1,549 pediatric scans it reports an MSE of 0.78 pixels and a 13.55% failure rate, against 0.86 and 24.19% for plain fine-tuning.

Both methods protect the pretrained representation. That is useful for keeping precision, but it also preserves the representation's normative bias. Our method preserves the pretrained representation too, but changes only its cross-midline communication.

Research Program

The first component is a symmetry-pull probe that needs no patient data. We take healthy faces from 300W and WFLW and apply a smooth, anatomically plausible warp to one hemiface only: lowering the mouth corner, widening the palpebral fissure, drooping the brow. The warped ground-truth landmarks are then known exactly.

For each detector we compute a tracking ratio, the fraction of the imposed displacement that the predicted landmarks follow. A ratio of 1 means the detector reports the deformation faithfully; a ratio below 1 means it pulls toward symmetry. We measure this for a convolutional hourglass/FAN-style detector, a PFLD-style regressor (PFLD), D-ViT and ORFormer, each trained on the same public data.

We also record the cross-midline attention mass per head in the two Transformers. We then test whether this mass predicts the per-image tracking deficit, both across heads and across images. For ORFormer we additionally record how often warped regions trigger the occlusion flag.

Synthetic warps are not real palsy, so the probe is repeated on real patients. On MEEI we compare signed asymmetry error between models with matched healthy-face NME.

The second component is the intervention. We use the stage-one landmarks of the frozen model to estimate the facial midline from the nose bridge, nose tip and chin. Each spatial token is labeled left or right by the position of its patch. Each channel token is labeled by whether its weight in the final heatmap projection falls mostly on left or right landmarks; channels that are ambiguous, for example those serving midline landmarks, are exempt. For head $h$, the attention logit between tokens $i$ and $j$ becomes

$$\tilde a^{h}_{ij} = a^{h}_{ij} + g_h \, c_{ij},$$

where $a^{h}_{ij}$ is the frozen model's logit, $c_{ij}\in\{0,1\}$ indicates that $i$ and $j$ lie on opposite hemifaces, and $g_h$ is a learned scalar initialized at 0. Initializing at 0 means the untouched model is recovered exactly, and a negative $g_h$ throttles cross-midline influence.

With D-ViT's heads across cascade stages, this amounts to a few dozen parameters. We pair the gates with the same small heatmap-head adapter used for every other adaptation baseline, so the comparison isolates the gates. We expect the gated model to learn its own balance. Heads that carry useful global context, such as pose and scale, should keep their gates near zero, while heads that impose left-right coupling should close.

The third component is the small-sample protocol. MEEI contains about 200 patients with three-clinician annotations. We split it by patient into a fixed test set and draw training subsets of 10, 25, 50 and 100 patients, with 5 random draws each. Palsy test sets A and B serve as external tests with a different annotation protocol; Set B has the stronger asymmetry.

Two further cohorts check that the method does not simply make models worse at symmetric-but-atypical faces. On the dementia cohort from Limitations and Biases in Facial Landmark Detection, gating should not be needed and should not hurt. On CIFASD, faces are dysmorphic but not lateralized, and we evaluate there if access is granted.

Baselines

The naive reference is the frozen, off-the-shelf D-ViT pretrained on WFLW and 300W, plus the same model after only a coarse landmark remapping to the MEEI landmark set. It shows how much symmetry pull exists before any adaptation. As a reference point, the original MEEI study found that its off-the-shelf model reached NRMSE 8.56 against 6.03 after retraining, in Toward an Automatic System for Computer-Aided Assessment in Facial Palsy.

We also re-run that paper's retrained regression-tree model on our patient splits. On 1,440 images it is the strongest published result on this repository, but it has not been reported at 10–100 patients. The only number we can quote is NRMSE 6.03 at the full training set. Our splits and protocol differ, so we compare only on re-run numbers.

The strongest learned competitor is RTL-SAM from Regularized Transfer Learning for FAS, which reports 0.78-pixel MSE and a 13.55% failure rate on CIFASD against 0.86 and 24.19% for plain fine-tuning. We use it for two reasons: it is the best documented small-sample landmark method in clinical faces, and it tests whether a strong CNN is enough without any attention. We run it on the same ResNet-50 initialization it used. Its regularization weight gets the same tuning budget as our method, 20 configurations chosen by patient-level cross-validation inside each training subset.

We also run Few-Shot Model Adaptation, which reports 4.71% NME on 300W with 5% of the data. It is the best documented frozen-backbone few-shot face method, and its frozen design matches ours in spirit.

On the hybrid itself we run three strong baselines under an identical tuning budget: full fine-tuning of D-ViT, fine-tuning of only its heads, and low-rank adapters on its attention layers. The last one isolates whether our gain comes from where we intervene or merely from adapting attention cheaply.

Two ablations isolate the key ingredient. In the first, gates are applied to randomly chosen token pairs with the same density as the cross-midline pairs; if this matches our method, the midline structure is not what matters. In the second, we hard-mask all cross-midline attention; if this matches the learned gates, learning is unnecessary but the mechanism is confirmed.

Beating the baselines means two things at each training size of 50 or fewer patients: a signed asymmetry error on MEEI significantly closer to zero than RTL-SAM and all D-ViT fine-tuning variants, with a paired test across the 5 draws, and healthy-face NME on the 300W test set that does not worsen by more than 0.1.

Why not just…?

Why not just retrain a simple detector on patient data, as Emotrics did? It works when about 1,440 images are available, as Toward an Automatic System for Computer-Aided Assessment in Facial Palsy showed. Rare conditions rarely offer that many annotated images, and regression trees have shown the largest errors on the most asymmetric cases in Atypical Facial Landmark Localisation. We run this baseline at every training size. If it matches our method at 10–50 patients, the conclusion is that hybrid models add nothing clinically, and we would report that.

Why not just use a CNN with regularized transfer and skip Transformers entirely? This is a genuine threat. Our own diagnostic might show that CNNs pull less toward symmetry. RTL-SAM, however, anchors the adapted model's spatial attention to a model pretrained on normal faces, which may preserve exactly the bias we measure. The symmetry-pull probe settles this directly. If RTL-SAM's tracking ratio and MEEI signed asymmetry error are as good as the gated hybrid's at equal data, the recommendation becomes "use regularized CNNs for asymmetric clinical faces," which is itself a useful, publishable negative result for hybrids.

Why not just fine-tune the hybrid, fully or with low-rank adapters? At 10–50 patients, full fine-tuning of attention parameters risks overfitting. The dementia study found that fine-tuning narrowed but did not close group gaps (Limitations and Biases in Facial Landmark Detection). Low-rank adapters are the fairer competitor because they are also cheap, and we give them the same tuning budget. If they reach the same asymmetry error, gates are only a convenience. Their value would then rest on interpretability and stability across the 5 draws, which we report separately.

Why not just augment the training data with synthetic one-sided warps and fine-tune normally? This alternative is the most likely to work, and the probe's own warps make it easy to build. We therefore include it as a baseline and as a combination: warp-augmented fine-tuning, with and without gates. If augmentation alone closes the gap on real MEEI patients and Test Set B, the architectural intervention is not needed and we would say so. We expect augmentation to help on the specific deformation types that were synthesized and less on unmodeled patterns such as synkinesis, and the per-region results on Set B test that expectation.

Proposed Experiments and Decisive Checks

The first decisive check is the probe. We need a measurable tracking deficit in D-ViT and ORFormer on synthetic one-sided warps. We also need a positive correlation between cross-midline attention mass and the deficit, computed per head and held out across images. Without it the premise fails and we stop at a diagnostic paper.

The second check is on real patients. On MEEI at 10, 25 and 50 patients, the gated D-ViT must bring signed asymmetry error on commissure excursion and eye opening closer to zero than every baseline in the previous section. Its gains must concentrate on the paretic side, and they must be present on Test Set B. Both the random-pair gate and plain low-rank adapters must fall short.

The third check guards the healthy case. Gating must leave NME on 300W and WFLW essentially unchanged, and it must not hurt the dementia cohort, where there is no lateral asymmetry to protect.

A secondary analysis asks whether ORFormer's occlusion flag fires on paretic eyes, measured against the eye-closure deficits documented in Automated analysis of spontaneous eye blinking in patients with acute facial palsy. A positive result would be a concrete warning against occlusion-repair models in clinical measurement.

The last analysis tracks the learned gate values across training sizes. Gates that consistently close the same heads would show that symmetry pull sits in a few identifiable parts of the network rather than spread through it.