Extrapolating Emotional Intensity in Speech-Driven 3D Faces

Speech-driven facial animation systems take an audio clip and produce a moving 3D face. They are trained on actors performing a handful of emotions at a few intensity levels. At test time they are asked to animate speech that is angrier, more tearful, or louder than anything they saw. Nobody currently measures what happens then. The best systems separate what is said, which drives fast, local lip and jaw motion, from how it is felt, which drives slower changes across brows, eyes, and cheeks. This separation is called content-emotion disentanglement. But these systems treat intensity as a category ("mild, medium, high") or as an unconstrained learned code, so nothing in the model says what "more intense than high" should look like.

This proposal borrows a lesson from out-of-distribution work on physical dynamics: models extrapolate when the parameter that shifts enters the model through a known, simple structure, and they fail when it is just another input. We propose a face model in which intensity is an explicit scalar that multiplies two learned emotion effects: a whole-face expression field, and an operator that reshapes articulation. The speech-content motion itself never sees the intensity value. We pair this with a test-time loudness normalization, so the content pathway does not collapse when intense speech arrives much louder than training speech. We also build the missing evaluation: an intensity-extrapolation split of MEAD and RAVDESS where the highest intensity is held out entirely.

Core Claim

If emotional intensity enters facial motion only through a fixed, low-order function of a supervised scalar, then a model trained on neutral, mild, and medium speech can predict high-intensity faces. It should do this with emotion error close to a model trained on all intensities, and without the lip-sync degradation that current content-emotion models show when pushed beyond their training range. The structure has two parts. The emotion effect is a whole-face displacement plus a low-rank modulation of articulation. The content motion is computed from loudness-normalized audio.

Concretely, let $x$ be the audio, $c_t = f(\tilde{x})_t$ the content-driven vertex displacement at frame $t$, computed from the loudness-normalized audio $\tilde{x}$, $e$ the emotion category, and $s \ge 0$ the intensity with neutral at $s=0$. The predicted face is

$$ y_t = c_t + \phi(s)\,\big(u_e(t) + A_e\, c_t\big) + r_t , $$

The terms are defined as follows. $u_e(t)$ is a slowly varying, emotion-specific displacement field. $A_e$ is a low-rank linear operator describing how emotion $e$ reshapes articulation, for example a wider jaw opening in anger. $r_t$ is a small residual network that receives no access to $s$. $\phi$ is a fixed-form curve fit through the three training anchors (neutral, mild, medium), either linear or a two-parameter saturating form. Which form is appropriate is itself an empirical question that we settle first.

Relation to Prior Work

The emotion-animation methods supply the representation and the baselines, but none targets extrapolation. EMOTE swaps categorical emotion and intensity labels during training and uses per-frame lip losses against sequence-level emotion losses, with a frozen temporal motion prior. It tests generalization to unseen speakers, not unseen intensities, and its three intensity tiers are discrete inputs with no defined value beyond "high." EmoTalk cross-reconstructs content from one clip with emotion from another and decodes to FLAME-style blendshapes. EcoFace aligns audio emotion features to visual motion with contrastive-triplet losses across continuous intensities. EcoFace is the closest in spirit, but its intensity lives in an unconstrained latent space with no guarantee about behavior outside the trained range. Cafe-Talk resolves conflicts between global emotion labels and local action units by swap-label training in a two-stage diffusion transformer. It also documents a relevant failure: networks trained jointly use side inputs as shortcuts for mouth shape. That warns us to keep $s$ out of the content pathway. ExpTalk separates motion and emotion codebooks. Our proposal differs from all five in one respect: intensity is a supervised physical-style parameter with a fixed functional role, and evaluation is on intensities absent from training.

The dynamics literature supplies the method. Disentangled Generative Models for Robust Prediction of System Dynamics supervises designated latent dimensions with known physical constants such as pendulum length. It shows this improves rollouts when those constants fall outside the training interval. We reuse the idea with intensity as the constant. Our addition is a restriction on how the parameter acts on the output, not only where it is stored. Graph-Operator World Models factorize transitions into a shared dynamics basis modulated by a morphology-conditioned operator. It reports that plainly feeding morphology as a context vector extrapolates poorly. Our $c_t$ plays the role of the shared basis and $\phi(s)A_e$ the operator. ExPhy shows that an explicit parameter interface plus a residual learned model generalizes to shifted parameters where purely data-driven forecasters collapse. This motivates our residual $r_t$, which is blinded to $s$. Buckingham π-Invariant Test-Time Projection treats multiplicative scale as a translation in log space and projects out-of-range test inputs onto the training range before running a frozen surrogate. We apply the same move to audio loudness, the most obvious multiplicative nuisance in shouted speech. That paper's method requires dimensional homogeneity, which speech lacks. We therefore use only its scale-translation step, and we test rather than assume that it helps.

Two papers constrain the design. Enforcing Analytic Constraints in Neural Networks Emulating Physical Systems shows that hard architectural constraints beat soft penalties for guaranteeing structure. It also shows they do not by themselves ensure accuracy under regime shift. Our factorization is exactly such a hard structure, so we must show that it predicts correctly, not merely that it behaves consistently. HOOD frames inputs as generated by content and style latents and synthesizes style-shifted examples for robustness. It motivates treating intensity as a style intervention on invariant content, though it works on static images and offers no mechanism for continuous extrapolation.

Research Program

The first step is a measurement, before any model is built. On 3D-MEAD (MEAD reconstructed to 3D meshes, as used by EMOTE), which has neutral plus three intensity tiers for eight emotions, we compute per-emotion mean deviations from neutral in the upper face and the mouth region at each tier. We test whether they scale affinely, saturate, or change shape. This tells us whether $\phi$ can be linear and whether the articulation term $A_e c_t$ is needed at all. If emotion at high intensity is not a scaled version of medium, the whole premise weakens, and we report that as the main finding.

The second step builds the model. The content network $f$ is a transformer over WavLM or wav2vec 2.0 features computed from loudness-normalized audio. Normalization translates the log-RMS envelope to the nearest cluster of training loudness, the audio analogue of the π-projection. The emotion field $u_e(t)$ is a low-dimensional temporal code decoded to vertices. $A_e$ is rank-constrained. Intensity $s$ is supervised to the ordinal anchors 0, 1, 2 during training. At test time $s$ comes from one of two sources: a user-specified value (the control regime), or a regressor on raw, unnormalized audio with a linear output head (the audio-driven regime). The regressor sees exactly the loudness information that the content path was denied. We keep EMOTE's lip-versus-emotion loss split, and we train with emotion swaps so that $c_t$ carries no emotion.

The third step is the benchmark. We create intensity-extrapolation splits: 3D-MEAD trained on neutral, mild, and medium and tested on high; and RAVDESS trained on normal intensity and tested on strong. Speakers are held out in both.

Baselines

The naive reference point trains the proposed backbone with intensity as an extra input vector and no structure. We evaluate it at the ordinal value 3. It isolates what the factorization adds over "just condition on $s$."

The strongest existing methods are EMOTE, EmoTalk, and a Cafe-Talk-style diffusion transformer. EMOTE is the most directly comparable because it already disentangles content from emotion with intensity tiers. EmoTalk is included because it reports the tightest vertex errors in the file: Lip Vertex Error 2.762 mm and Emotional Vertex Error 2.493 mm on RAVDESS, in distribution, against FaceFormer's 3.247 mm LVE. Cafe-Talk reports 59.48% emotion classification accuracy on MEAD. We found no reported result on Lacuna for any method on a held-out-intensity split, so these numbers anchor only the in-distribution regime. All baselines are retrained on our extrapolation splits with identical data, the same speech encoder, matched parameter count, and the same hyperparameter search budget. Each method's highest trained setting stands in for "high."

The oracle is the proposed model and EMOTE trained on all intensities. It bounds what extrapolation could achieve.

Ablations remove, one at a time, the loudness normalization, the articulation operator $A_e$ (leaving an additive emotion field only), the blinding of $r_t$ to $s$, and the supervision of $s$.

Beating the baselines means two things on the held-out tier. First, Emotional Vertex Error must be lower than every retrained baseline and closer to the oracle than to the best baseline. Second, Lip Vertex Error must be no worse than the same model's error on the medium tier.

Why not just…?

Why not just linearly extrapolate an existing model's outputs, taking medium plus (medium minus mild)? This is a real threat. If EMOTE's mild and medium outputs differ only in the emotion component, output-space extrapolation is nearly our model without training changes. The risk is that black-box models also alter lip shapes between tiers, so extrapolating amplifies viseme distortion. We run this baseline on every retrained model. If it matches ours on both EVE and high-tier LVE, the contribution reduces to the benchmark and the finding that existing models are already affine in intensity, and we would say so.

Why not just raise the classifier-free guidance scale of a diffusion model? Guidance pushes samples further along the conditioning direction, which is a form of intensity extrapolation. However, it amplifies the entire conditional signal, including any speech-emotion interaction, and it is known to oversaturate. We sweep the guidance scale on the Cafe-Talk-style baseline. We select the scale on the medium tier, since choosing it on the test tier would leak the answer. We then compare at the chosen scale and also report the full LVE-EVE tradeoff curve against ours.

Why not just train on more in-the-wild data that already contains intense speech? Cafe-Talk's 157-hour internet training still showed shortcut failures. ExPhy and the SD-VAE study both show that broad data does not cover the tail of a shifting parameter. The practical issue is also that intense, well-reconstructed 3D speech is exactly what is scarce. The oracle comparison tells us how much held-out-tier data would be worth relative to structure.

Decisive Checks

The primary check is emotion error and lip error on the held-out high tier of 3D-MEAD and on strong RAVDESS. We add emotion-classification accuracy and an intensity regressor applied to the generated meshes, and lip-sync distance on rendered video for the audio-driven regime. The factorization is supported if its high-tier emotion error closes most of the gap to the oracle while high-tier lip error stays flat, and linear output extrapolation and guidance scaling do not. The loudness ablation is decisive for the audio-driven regime. If removing normalization raises high-tier lip error but leaves emotion error unchanged, the two roles of loudness are genuinely separable.

One caveat is that all meshes are pseudo-3D reconstructions, so small differences near reconstruction noise should not be over-read. We report per-emotion results and confidence intervals across held-out speakers. A negative outcome is also informative. If the preliminary measurement shows high intensity is not a low-order function of mild and medium, then intensity extrapolation from categorical data is ill-posed, and the field's continuous-intensity claims need new data rather than new models.