A person's face moves in a recognizable way. The timing of a smile, how fast the lip corners pull back, and how the brows rise during speech carry identity information even after you discard what the face looks like. Systems that exploit this already work well at a yes/no level. A small network compares how a test video moves against enrolled videos of the claimed person and outputs one similarity score. Examples are deciding whether the person driving a synthetic talking-head avatar is authorized, or whether a video of a politician is a reenactment. That score says nothing about why the clip was rejected. In forensic settings, a decision nobody can inspect is hard to act on.
This note proposes a specific way to attach a language model to such a system so that it produces audits that can be checked. We freeze an appearance-blind motion encoder. We split its output into one short sequence of continuous vectors per facial region, and feed those vectors through a small learned projector into a frozen large language model (LLM). The LLM never makes the accept/reject decision; the frozen metric head keeps that job. The LLM's only job is to say which region, which moment, and what kind of motion deviates, for example "onset velocity of the left lip corner is too fast in 0.8–1.3 s." Every such statement is scored against ground truth that we build ourselves by splicing one person's regional motion into another person's clip. The question is whether continuous projection gives explanations that are more faithful and more specific than the obvious non-LLM alternative, which is reporting per-region embedding distances.
Core Claim
The claim has two parts. The first is that continuous region-level motion tokens, projected into a frozen LLM, support temporally localized and typed explanations of identity mismatch. "Typed" means the explanation says whether the deviation is in amplitude, timing, velocity, or asymmetry. The second is that these explanations are causally faithful to the evidence the verifier uses. We test faithfulness by deleting the tokens of the region the LLM names: this should move the metric head's score more than deleting any other region's tokens. A per-region distance readout can localize where a mismatch is. It cannot localize when or say what kind without hand-built rules. That gap is where the LLM has to earn its place. If the distance readout plus simple rules matches it, the claim fails and we will report that.
This sits in a gap the prior work leaves open. Motion-based verifiers already separate identity from appearance but output only scores. Avatar Fingerprinting uses pairwise landmark distances and a temporal 3D convolutional network over 71-frame clips, reaching 0.85 average AUC on authorized-versus-unauthorized avatar driving. ID-Reveal learns a temporal identity network over 3D face-model coefficients, adversarially hardened, and improves by more than 15% over pixel detectors on heavily compressed reenactments. We reuse their input representations and keep their verdict, adding a reasoning layer neither has.
The closest LLM work, MELLM, does put facial motion into an LLM. It does so by binning optical flow into text descriptions of 29 regions, for affective micro-expression understanding, with no identity component. We test its text-serialization route as a direct competitor rather than assuming continuous projection wins. DFER-CLIP aligns video with language descriptions of muscle movements, but only contrastively and only for expression categories pooled across people. General video LLMs such as Video-LLaVA sample 8 frames per clip, which is too coarse for 71-frame motion signatures. MELLM reports that general multimodal LLMs miss subtle facial motion in 25–70% of cases. Both findings constrain the design: the visual pathway must not be a generic frame encoder.
Research Program
Input representation. We start from appearance-stripped inputs: pairwise landmark distances as in Avatar Fingerprinting, and per-frame 3D morphable model (3DMM) expression and pose coefficients as in ID-Reveal. A 3DMM is a parametric face model whose coefficients separate shape, expression, and head pose. Landmarks are grouped into roughly ten anatomical regions: each brow, each eye, nose, each lip corner, upper lip, lower lip, and jaw.
Motion encoder. A temporal encoder per region is trained with a contrastive identity objective and the time-shuffle penalty from Avatar Fingerprinting. A shared metric head on the pooled output gives the verdict. Then everything is frozen.
Projector. A small perceiver-style projector compresses each region's 71-frame trajectory into two or three tokens. A perceiver block is a few learned query vectors that attend over a long sequence and summarize it. The projector adds learned region and clip-role embeddings that mark each token as reference or probe. For an enrollment of three reference clips plus one probe, this keeps the LLM input under about 120 tokens. That answers the token-explosion problem by compressing in time per region, unlike MiniGPT-v2, which concatenates neighboring tokens in space.
Training regime. Following Scalable Vision Language Model Training via High Quality Data Curation, only the projector is trained in the first stage. Both the encoder and the LLM stay frozen, which protects the LLM's language ability; Deep Pre-Alignment for VLMs documents MATH-500 falling from 84.8 to 36.4 under standard multimodal tuning. An optional second stage adds low-rank adapters, a small set of trainable weights inserted into the frozen LLM. That stage is reported separately.
Supervision from controlled splices. For a genuine probe clip, we replace one region's trajectory over a chosen time window with the time-aligned trajectory of another identity speaking the same script. NVFAIR's scripted monologues make this possible. We optionally perturb the splice in one controlled way: time-scaling for timing, gain for amplitude, or one-sided application for asymmetry. The ground-truth audit (region, window, deviation type) is known by construction. Templated target text is generated from it, so the LLM learns to read the tokens, not to invent rationales. Genuine and impostor pairs without splices teach it to say "no localized deviation" or "insufficient motion." The latter matters for the low-expressivity subjects that Avatar Fingerprinting flags as a failure mode.
Baselines
Naive reference. The do-nothing point is the frozen metric head alone, with no explanation, plus a random-region, random-window "explanation." This sets the floor for every localization metric. It also confirms that verification AUC is untouched, since the verdict path is shared by construction. For context, Avatar Fingerprinting reports 0.85 average AUC on NVFAIR and 0.83 on unseen generators. Our re-trained head should land within 0.02 of those numbers before any explanation result is trusted.
Strongest non-LLM competitor: region distance plus rules. On the same frozen encoder, we compute the reference-to-probe distance for each region and pick the top region. We then add the best rule-based localizer we can build: a sliding-window distance over each region's per-frame features, plus hand-written rules for deviation type. The rules compare velocity, amplitude, and left-right asymmetry statistics against the enrollment. This is the real threat, because it uses identical evidence and zero LLM. Its window size and thresholds get the same tuning budget on the same validation split as the projector. Beating it means higher region top-1 accuracy, higher temporal overlap (intersection-over-union) of the predicted window, and higher deviation-type accuracy on held-out identities and unseen generators. We set a margin of at least five points on the latter two before we claim an advantage.
Strongest existing LLM route: MELLM-style serialization. Following MELLM, we serialize per-region motion statistics (mean and maximum magnitude, dominant direction, and the window statistics used by the rule baseline) into structured text. That text is fed to the same LLM, with the same adapter budget and the same splice training data. No published number exists for any LLM-based identity audit; there is no reported result on Lacuna for explanation faithfulness on NVFAIR or FF++.
Ablation that isolates the key ingredient. We replace the pretrained LLM with a randomly initialized transformer of matching size, trained with the same projector and data. This is the decisive test of whether language pretraining contributes anything beyond a sequence-to-template mapper. If the random transformer matches the frozen pretrained LLM on held-out generators and on real reenactments, the language model is not doing the work.
Why not just…?
Why not just report per-region distances from the metric head? For "which region," this may be fully sufficient, and we expect it to be competitive. It is the main baseline above, run on identical features. The proposal's claim rests on the two things distances do not give for free, timing and deviation type, and on transfer to out-of-distribution cases. If the tuned rule-based localizer matches the LLM on those too, the honest conclusion is that forensic audits need structured distance reports, not language models. That outcome is still useful, and we will publish it.
Why not just serialize motion into text, as MELLM does? Text binning is cheap and keeps the LLM in its native input space. However, it fixes in advance which statistics survive, and identity-bearing speech dynamics are known to live in continuous trajectories. MKPLS separates speaker identity from phonetic content only by projecting full spatiotemporal trajectories into subspaces. Whether that argument matters for audits is empirical. The matched serialization baseline settles it, especially on splices whose only change is a velocity profile with equal mean magnitude. We build those specifically because binning should miss them.
Why not just fine-tune a general video LLM on the splice data? A Video-LLaVA-style model sees 8 frames of RGB. That is too sparse in time, and its input carries appearance, which invites the demographic and environmental shortcuts documented by Learnable PINs. We include it once as a reference point on NVFAIR. In NVFAIR, a single avatar appearance can be driven by many people, so appearance carries no information about who is driving. A model relying on appearance should therefore fail. If this model unexpectedly succeeds, that would show the frame pathway carries more motion than MELLM's failure rates suggest.
Decisive Experiments
Datasets. The primary benchmark is NVFAIR, with 161 identities across three generators. Two generators are used for training, and the third is held out for zero-shot tests matching Avatar Fingerprinting's protocol. Identities are split disjointly. Splices are generated within the scripted monologues. Natural conversations are used only for testing, without splices, to measure false "deviation" claims on genuine pairs.
Real manipulations. The transfer test uses FaceForensics++ Face2Face reenactments at heavy compression, scored with ID-Reveal-style references. FaceForensics++ reports XceptionNet at 81.00% accuracy on this low-quality setting. ID-Reveal reports pixel detectors near 50–60% and its own result more than 15% higher, with no single absolute figure given in the file. On FF++ we have no ground-truth localization. Instead, we test whether the LLM's named region and window coincide with the region the face-model expression transfer actually alters, the mouth and jaw. We also apply the same token-deletion faithfulness check.
Longitudinal check. 4DFAB, where 3DMM identity recognition reaches 96% across sessions spanning five years, tests whether audits stay stable across sessions for genuine pairs. The measure is low false-deviation rate when one subject's sessions are compared.
Metrics. We report verification AUC (unchanged by design, checked anyway), region top-1 accuracy, temporal intersection-over-union, and deviation-type accuracy. We also report the causal faithfulness gap: the drop in metric-head score from deleting the named region's tokens minus the mean drop from deleting other regions' tokens. Finally, for abstention, we report calibration of "insufficient motion" against a motion-energy threshold baseline.
What would settle the question. The proposal succeeds if the frozen pretrained LLM beats both the tuned rule localizer and the serialization baseline on timing and type for the held-out generator. It must also show a positive faithfulness gap on FF++ and outperform its randomly initialized twin. It fails cleanly if any one of these three comparisons is a tie.
References
- Avatar Fingerprinting for Authorized Use of Synthetic Talking-Head Videos
- ID-Reveal: Identity-aware DeepFake Video Detection
- MELLM: Exploring LLM-Powered Micro-Expression Understanding Enhanced by Subtle Motion Perception
- Prompting Visual-Language Models for Dynamic Facial Expression Recognition
- Video-LLaVA: Learning United Visual Representation by Alignment Before Projection
- MiniGPT-v2: Large Language Model as a Unified Interface for Vision-Language Multi-task Learning
- Scalable Vision Language Model Training via High Quality Data Curation
- Deep Pre-Alignment for VLMs
- MKPLS: Manifold Kernel Partial Least Squares for Lipreading and Speaker Identification
- Learnable PINs: Cross-modal Embeddings for Person Identity
- FaceForensics++: Learning to Detect Manipulated Facial Images