The best recent methods for recognizing facial expressions on datasets they were never trained on share one recipe. They keep a large web-trained image model frozen and learn only a small head on one labelled source dataset. The leading example, CAFE, freezes CLIP (an image–text model trained on 400 million web image–caption pairs) and reports 73.16% accuracy when transferring from RAF-DB to FERPlus. The authors attribute this to CLIP's general knowledge of faces. There is a second explanation nobody has ruled out: RAF-DB, FERPlus, SFEW and AffectNet have been publicly hosted on the web for years, so CLIP may simply have seen the test images. In language models, this kind of test-set leakage ("benchmark contamination") is known to inflate scores badly. Vision has no tool for measuring it, and for CLIP the training corpus is not even public, so the leaked images cannot be searched for.
This note proposes a way to measure it. For every test image in a benchmark, we generate a synthetic twin: a new face showing the same expression, with the same muscle movements, head pose and lighting, but a different identity and entirely new pixels. The twin keeps the original human label. It comes from a generator whose training data has been scrubbed of every evaluation benchmark. A model that understands expressions should lose only a little accuracy going from originals to twins. A model that partly memorized the originals should lose more. Synthetic images are imperfect, so every model loses something on twins. We subtract the loss suffered by matched reference models that provably never saw the benchmarks. What remains is an estimate of how much of the benchmark score is memory rather than generalization.
Core Claim
Contamination in frozen-foundation facial analysis can be measured without access to the pre-training corpus. The method is a difference-in-differences: compare the accuracy drop from real benchmark images to their synthetic twins for the model under test with the same drop for a contamination-free model of the same architecture.
Let $a_M(\text{real})$ and $a_M(\text{twin})$ be the accuracy of a model $M$ on the original test images and on their twins. Let $R$ be a reference model with the same architecture, trained on a corpus verified to exclude the benchmark. The contamination estimate is
$$ \hat{C}_M = \big[a_M(\text{real}) - a_M(\text{twin})\big] - \big[a_R(\text{real}) - a_R(\text{twin})\big]. $$
The second bracket absorbs the synthetic-to-real gap: smooth skin, diffusion artifacts and missing micro-texture. The first bracket contains that gap plus whatever the model gains from having seen the specific originals.
Because each twin inherits its source image's label, annotator subjectivity is held fixed. That subjectivity is the dataset-specific shift in which categories annotators assign, documented by A Deeper Look at Facial Expression Dataset Bias. Holding it fixed matters, because label disagreement is the usual excuse for cross-dataset drops. Our claim is falsifiable. If $\hat{C}$ for CLIP-based CAFE on FERPlus is indistinguishable from zero under a test that demonstrably detects injected contamination, CAFE's generalization claim survives a test it has not yet faced.
The three research directions play different roles here. Cross-dataset facial expression recognition is the application and the claim under scrutiny. Benchmark contamination is the question. Synthetic-to-real transfer is the measurement instrument, and also the main confound that the design must cancel.
Relation to Prior Work
Two papers motivate the work by showing the gap.
- Generalizable Facial Expression Recognition introduces CAFE: frozen CLIP ViT-B/32, a ResNet-18 branch producing channel masks, and per-emotion channel groups with max pooling in place of a fully connected head. It reports a 63.48% mean over four unseen targets, against 54.88% for the prior best (RUL). It never asks whether CLIP saw those targets. We do not propose a better classifier; we test whether this one's numbers mean what they claim.
- Does CLIP Know My Face? shows that CLIP retains information about specific faces from its web data. That makes memorization of benchmark faces plausible. However, it studies identity, not expression benchmarks, and offers no evaluation protocol.
Language-model contamination work supplies the logic but not the method.
- DyVal procedurally generates fresh reasoning problems. Models strong on static benchmarks (phi-1.5, WizardMath-13B) score near zero on structurally equivalent fresh items.
- Detecting Benchmark Contamination Through Watermarking plants token-level watermarks before release. It reports detection at $p=10^{-3}$ for a 5% accuracy inflation on ARC-Easy.
Both work on text. Watermarking also only helps for benchmarks released after the watermark, which excludes every face benchmark CLIP could have scraped. DyVal's idea of fresh items with matched structure is what we carry into pixels. Our structure-preserving variable is the facial muscle configuration rather than a reasoning graph.
The synthetic-face work supplies the method and also constrains it.
- SynFER conditions a diffusion model (a generator that turns noise into images by repeated denoising) on facial action units. Action units are the standard coded muscle movements, such as "brow lowerer" or "lip corner puller." Adding them raised expression faithfulness from 34.62% to 48.74%. We reuse this conditioning. But SynFER's training set includes AffectNet and SFEW images, so its generator is itself contaminated. We turn that flaw into an ablation.
- AU-Supervised Convolutional Vision Transformers for Synthetic Facial Expression Recognition shows that action-unit supervision helps synthetic-to-real transfer. It also shows that pseudo action-unit labels come from a detector trained on real faces, which is a dependency we must audit too.
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks found that pixel translation handles texture and lighting but not structural change. This is why we build twins by re-synthesizing from action units and pose rather than by style-transferring the original. Style transfer would leave the original's geometry, and therefore its memorizable signature, in place.
Research Program
Sanitized generator. We assemble a face corpus from FFHQ and CelebA-HQ, plus any further licensed face data. We then remove near-duplicates of every image in RAF-DB, FERPlus/FER2013, SFEW 2.0, AffectNet, MMA, CK+ and JAFFE, using perceptual hashing and face-embedding nearest-neighbour search, with a conservative threshold. We train an action-unit-, pose- and lighting-conditioned diffusion model on this corpus.
Twin construction. For each benchmark test image, we extract action-unit intensities, head pose and a coarse lighting estimate. We then sample a twin with a new identity under those conditions. Twins enter the test set only if three checks pass:
- the twin's re-extracted action units match the source within tolerance;
- an identity matcher confirms the twin is a different person;
- on a stratified sample of about 2,000 twins, crowd raters agree with the inherited label at a rate comparable to the original images' rater agreement.
The action-unit detector is an instrument, not a model under test. Its own training data, possibly including CK+, is reported, and CK+ is treated as a lower-confidence target.
Reference models. The design rests on references, and building them is the costliest part. We train ViT-B/32 CLIP models from scratch on a LAION subset of a few tens of millions of pairs, with the same near-duplicate filter applied. This is the clean reference $R$. Then we build dose-response variants: the same run with benchmark test images and captions injected at 1, 4 and 16 exposures. Cheaper secondary references are also included:
- a face-recognition-pretrained ResNet-50 with documented training data;
- a MoCo v3 encoder pre-trained on the sanitized corpus.
Models audited. CAFE on OpenAI CLIP, and CAFE on OpenCLIP checkpoints trained on public LAION. The LAION checkpoints matter because their corpus can be searched directly, which provides independent ground truth.
Baselines
The naive reference point is the raw real-to-twin drop for CAFE, ignoring reference models. It is what a reader would compute first. We expect it to be large and uninterpretable, because synthetic faces lose accuracy for every model. We report it to show how much of it the reference subtraction removes.
The strongest existing system on the underlying question is CAFE itself, trained on RAF-DB exactly as in Generalizable Facial Expression Recognition. We first reproduce its reported 63.48% mean accuracy and 73.16% FERPlus accuracy on the original test sets. Our reference CLIP models then get the identical CAFE head, training data, schedule and hyperparameter search, so that $a_R$ and $a_M$ differ only in the backbone's pre-training data. "Beating" CAFE does not mean a higher accuracy here. It means showing that a measurable share of its accuracy, with a bootstrap confidence interval excluding zero, does not survive twinning beyond what the clean reference loses. Alternatively, it means certifying that no such share exists.
The strongest existing detection baseline is direct near-duplicate retrieval of benchmark images in the pre-training corpus. This is the visual analogue of the n-gram overlap audits surveyed in Benchmark Data Contamination of Large Language Models: A Survey. We run it on LAION for the OpenCLIP checkpoints. No reported result was found on Lacuna for retrieval-based contamination rates of facial expression benchmarks in any image–text corpus, so this number is itself an output of the project. A second detection baseline is a membership-style score: CLIP similarity between a test image and an emotion caption, compared between originals and twins without the reference correction. The twin test beats these baselines if two things hold. First, its estimates track the injected dose in the controlled runs. Second, on OpenCLIP it agrees with retrieval-measured overlap where retrieval is possible.
Two ablations isolate the key ingredients. The first rebuilds twins with a generator trained on a SynFER-style corpus that includes AffectNet and SFEW. If that generator's twins of AffectNet images shrink CAFE's measured contamination, the generator's own leakage is masking the effect. That would directly quantify the open problem in SynFER. The second keeps the source identity and changes only pixels, by re-sampling noise. Comparing it with full twins separates identity memory from image memory.
Why not just…?
Why not just search the pre-training data for the benchmark images? For OpenAI CLIP the corpus is unreleased, so this is impossible, and that is the most-cited model in this line of work. For OpenCLIP it is possible, and we do it. It serves as validation, not as a competitor. Even where retrieval works, it only shows presence, not whether presence changed accuracy. The watermarking paper reports that models often fail to memorize isolated canaries while still absorbing distributed signals (Detecting Benchmark Contamination Through Watermarking). So the count of copies and the effect on the score can come apart.
Why not just collect a fresh real test set after CLIP's cutoff? This is the most serious alternative. For a single new dataset it would be cleaner than synthesis, and we include a small fresh set of a few thousand consented, newly photographed images. Fresh data, however, changes the annotators. A Deeper Look at Facial Expression Dataset Bias shows that each dataset's labelling has its own signature, and a simple classifier identifies the source dataset 79% of the time against roughly 14% chance. A drop on fresh data therefore confounds contamination with label shift. Twins keep each benchmark's labels while changing every pixel, and they can be built for all seven benchmarks at once. If twin-based and fresh-data-based estimates agree on the shared target, the method is validated. If they disagree, the reference-matched drops on the fresh set show which confound is responsible.
Why not just compare CAFE against a linear probe or an end-to-end fine-tuned model? Head comparisons say nothing about the backbone's data. CAFE's paper already shows that fully connected heads overfit source quirks, and both heads sit on the same possibly-contaminated features. Only varying the pre-training data while holding architecture and head fixed answers the question. That is why reference CLIPs with the same architecture are part of the design.
Proposed Experiments and Decisive Checks
The first check is calibration on the controlled-injection CLIPs. $\hat{C}$ must be near zero for the clean reference. It must also rise monotonically with injected exposures (1, 4, 16) on RAF-DB, FERPlus and SFEW 2.0 test images. If the twin test cannot detect contamination we planted ourselves, nothing it says about OpenAI CLIP counts.
The second check is the main audit. We compute $\hat{C}$ for CAFE on OpenAI CLIP and on OpenCLIP across FERPlus, AffectNet, SFEW 2.0 and MMA, training on RAF-DB as in the original protocol. We report bootstrap confidence intervals over test images and over three generator seeds. We stratify by action-unit intensity, because subtle expressions should rely more on memory if memory matters.
The third check is validity. We require four things:
- human label agreement on twins comparable to that on originals;
- agreement between $\hat{C}$ and LAION retrieval overlap for OpenCLIP;
- agreement with the fresh real set;
- the generator-contamination ablation behaving as predicted.
The decisive outcomes are stated in advance. Suppose OpenAI-CLIP CAFE shows a contamination estimate whose interval excludes zero on FERPlus, at a magnitude the calibration runs associate with real exposure. Then part of the reported zero-shot generalization is memory, and the field's benchmark protocol needs twinned test sets. Suppose instead the estimate is null while the calibration runs show clear sensitivity. Then CAFE's generalization claim has passed the first contamination test applied to it. Either way, we release the twin sets and the sanitized generator so future frozen-backbone methods can be audited the same way.
References
- A Deeper Look at Facial Expression Dataset Bias
- Generalizable Facial Expression Recognition
- Does CLIP Know My Face?
- DyVal: Dynamic Evaluation of Large Language Models for Reasoning Tasks
- Detecting Benchmark Contamination Through Watermarking
- SynFER: Towards Boosting Facial Expression Recognition with Synthetic Data
- AU-Supervised Convolutional Vision Transformers for Synthetic Facial Expression Recognition
- Unsupervised Pixel-Level Domain Adaptation with Generative Adversarial Networks
- Benchmark Data Contamination of Large Language Models: A Survey