Reading Long Facial Videos Like Immune Repertoires

A person's immune repertoire and a long clinical video of their face present the same machine-learning problem. The repertoire is the full set of immune receptor sequences in a blood sample. The video is the stream of facial muscle activity across an interview. In both cases, the label we have describes the whole person ("CMV-positive", "depressed", "genuinely amused"). The evidence for that label sits in a tiny fraction of the parts: 0.01–0.1% of receptor sequences, or a few hundred milliseconds of frames in an hour of footage. Worse, most of the remaining parts are not neutral noise. They are common patterns that people in both classes share. In immunology these are "public" receptors present in nearly everyone. In faces they are social smiles, speech-related lip movements, and blinks.

Immune repertoire researchers built tools for exactly this setting. Facial action unit researchers have not used them. We propose to bring three of these tools into long-video facial analysis. The first is attention pooling designed to retrieve a handful of relevant items from huge sets. The second is a step that relabels items in positive cases as "background" when they look like what negative cases also contain. The third is weighting each item by how often it recurs. The facial representation we apply them to is facial action units (AUs): standardized codes for individual muscle movements, such as AU6 for cheek raising around the eyes and AU12 for pulling the lip corners. We then test the reverse transfer as well: whether the person-specific correction used for faces helps repertoire classification.

Core Claim

Weakly labeled long facial videos have a structure close to immune repertoire classification, and the tools designed for repertoires should beat standard video pooling under specific conditions. Those conditions are that the diagnostic signal is rare, that its form is not known in advance, and that look-alike patterns occur in both classes. The immunology side supplies the method. Modern Hopfield Networks and Attention for Immune Repertoire Classification (DeepRC) treats a repertoire as one huge bag of instances and pools them with attention viewed as associative-memory retrieval. It reports 0.984 AUC on simulated repertoires where only 0.1% of sequences carry the signal. A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach for Immune Repertoire Classification (LaDM³IL) adds class prototypes. These stop the model from assuming that every instance in a positive bag is disease-related, and the model aggregates instance scores weighted by clonal frequency. The facial side supplies the motivation. Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods describes genuine and posed expressions that differ only in co-activation and timing, such as AU6 together with AU12 versus AU12 alone. Facial Action Unit Event Detection by Cascade of Tasks frames AU detection as finding sparse events in time.

The research file found no work connecting the two fields. It lists the cross-domain transfer of repertoire-style multiple instance learning to video only as an open gap. Our proposal differs from all cited facial work, which trains frame- or clip-level detectors with frame-level labels. We learn from video-level labels alone, and we treat discovering the rare frames as part of the output.

Research Program

Each video becomes a bag of instances. An instance is a short window of frames, encoded as a vector. The vector combines AU intensities from a frozen AU encoder with simple kinematic features: onset velocity, apex duration, and left–right asymmetry. These kinematic cues are the ones the genuine-versus-posed literature identifies as diagnostic. The encoder is a transformer-based AU detector in the style of Facial Action Unit Detection With Transformers, trained on BP4D and then frozen, so every compared method sees identical instance features. The pairing of AU configuration with kinematics plays the role that LaDM³IL's fusion of CDR3 sequence with V/J gene identity plays for receptors: two kinds of description of the same item, combined multiplicatively.

Pooling follows DeepRC. A small set of learned queries retrieves instances through softmax attention over the whole bag. A large inverse-temperature parameter lets one query lock onto a few windows among tens of thousands. Next we add LaDM³IL's disambiguation step. It maintains a running prototype for "negative-class content" and one for "positive-class-specific content". Windows in positive videos whose embeddings sit closer to the negative prototype are reassigned to background before pooling. This is the step aimed at the social-smile problem: a smile that every participant produces should stop counting as evidence. LaDM³IL weights receptors by clonal frequency. Its facial analogue is recurrence. Windows are clustered within a video, and a configuration the person repeats several times is weighted above an isolated detector blip. That weighting helps reject tracking artifacts.

Finally we borrow from facial personalization. Selective Transfer Machine for Personalized Facial Action Unit Detection reweights training data so its distribution matches each unlabeled test subject. In the same spirit, we normalize each window against that person's own background distribution, computed from the person's own video, before prototype assignment. Strong AU12 in someone who smiles constantly is then less informative than the same AU12 in someone who rarely does. The repertoire literature motivates the reverse version of this correction. Identification of a type 1 diabetes–associated T cell receptor repertoire signature from the human peripheral blood shows that genetically restricted, person-specific receptors dominate background. So we test the person-specific correction back on the repertoire benchmark.

Baselines

The naive reference averages frame-level AU detector outputs over the whole video and fits logistic regression. This is what most digital-phenotyping pipelines effectively do. Beating it shows only that the rare signal matters.

The strongest simple competitor is top-k pooling over the same frozen AU features. It keeps the k highest-scoring windows under a learned linear scorer, tunes k, and classifies from the pooled vector. This competitor matters because it already captures "rare but strong" evidence without any repertoire machinery.

The second competitor is gated-attention multiple instance learning: standard soft attention over windows with no associative-memory sharpening and no disambiguation. It is the generic method a video researcher would reach for.

The third competitor is DeepRC run unchanged on the facial bags, which isolates the value of our additions. On the facial side, a frame-supervised event detector in the spirit of Facial Action Unit Event Detection by Cascade of Tasks serves as an upper reference. It gets frame-level labels where those exist, which our method does not.

All methods share the frozen encoder, the window definition, the subject-disjoint splits, and an equal hyperparameter search budget of 50 trials per method on validation data. All are scored by video-level ROC-AUC with subject-level bootstrap intervals, and by instance-level AUC for recovering planted or annotated windows.

The research file contains no reported number for any video-level multiple-instance task on facial data, so no reported result found on Lacuna for weakly supervised long-video AU classification. The reference points that do exist are on repertoires. DeepRC reports ROC-AUC 0.831 on the CMV cohort and 0.984 on 0.1%-witness simulations. LaDM³IL reports 84.88% AUC on CMV and 0.767 AUC for recovering 164 known CMV-associated receptors. We will count the proposal as successful on faces if it beats top-k pooling by a bootstrap-significant margin. That margin must hold at witness rates of 0.1% and below, and specifically under the confounded condition described in the experiments.

Ablations remove, one at a time, disambiguation, recurrence weighting, the person-specific normalization, and the associative-memory sharpening (by replacing it with gated attention). The key ingredient we expect to matter is disambiguation. If removing it costs nothing, the core claim fails.

Why not just…?

Why not just use top-k pooling over a good AU detector? This is the real threat. If the diagnostic pattern is a known, fixed configuration, top-k on the right AU score, or even a hand rule for AU6 plus AU12, should match anything we build. We do not argue around this. The confounded synthetic condition plants the same AU12-dominant windows in both classes and makes only the kinematic variant class-specific. If top-k still matches our method there, we conclude the repertoire tools add nothing for faces and report that. Our prediction is that top-k picks the strongest smiles, which are often the shared social ones. Disambiguation is supposed to fix exactly that failure.

Why not just train a frame-level detector for the target pattern? That requires knowing the pattern and labeling frames. In clinical recordings such as hypomimia, craving, or schizophrenia interviews, the label is a diagnosis or a symptom score, and which moments matter is unknown. The GNN study in Facial AU-aid hypomimia diagnosis based on GNN reached 91.7% accuracy because it used a structured smiling task that concentrates the signal. Our setting is unstructured recording. The frame-supervised reference in our baselines measures how much is lost by not having frame labels.

Why not just use temporal action localization or a video transformer end to end? Memory limits end-to-end training over hour-long bags, the same limitation the repertoire papers note. Recent end-to-end models also have no mechanism for down-weighting content shared across classes. We include gated attention as the cheap stand-in for this family. If it matches our method under confounding, the disambiguation claim fails.

Proposed Experiments

The decisive experiment is a planted-witness benchmark modeled on DeepRC's simulated repertoires. We take long spontaneous recordings from DISFA and BP4D as background. Into positive videos we splice short windows containing a target configuration with a specific kinematic profile, taken from genuine-expression material. We sweep the witness rate from 1% to 0.01%. In the confounded condition, negative videos receive matched-intensity look-alike windows that differ only in timing or AU6 involvement. The outcomes are video-level AUC and witness-recovery AUC across this sweep. The prediction is that our method separates from top-k and gated attention only in the confounded, low-witness corner. If the separation shows up everywhere, we would suspect a leak in the benchmark rather than credit the method.

On real data, we use genuine-versus-posed corpora reviewed in Detection of Genuine and Posed Facial Expressions of Emotion: Databases and Methods. Labels are applied at the session level, and we hide the frame-level annotations, which serve only for evaluating window recovery. We also use the confusion corpus from Multimodal Modeling of Task-Mediated Confusion and the REC dialogue corpus from The Modulation of Cooperation and Emotion in Dialogue: The REC Corpus, where labels are coarse and expressive events are sparse. A qualitative check is to ask FACS coders, blind to method, whether the retrieved windows are the ones they would cite.

The reverse-transfer check runs the person-specific normalization inside LaDM³IL on the CMV cohort. Each repertoire's sequences are reweighted against that individual's own background distribution before prototype assignment. Success means exceeding the reported 84.88% AUC under the same splits, together with improved recovery of the 164 known CMV receptors over the reported 0.767. A null result would still be informative, because it would suggest that HLA-driven private background differs in kind from personal facial style.