A talking-head avatar that follows a typed instruction such as "speak with a deep frown, growing angrier" has to do three things at once. It must change the face in ways that geometry alone cannot express, such as forehead creases, nasolabial folds and brow furrows. It must keep those details stable from frame to frame. It must render fast enough for live conversation. Current systems manage two of these. 3D Gaussian Splatting represents a head as many small colored, semi-transparent blobs that are drawn directly to the screen, and it reaches 98 to 300 frames per second. The fastest of these systems stay stable because they never change a blob's color or opacity: they only move it. Systems that do change appearance to render text-driven wrinkles generate those changes per frame with an iterative denoiser, so they pay in both speed and stability.
This note proposes a middle path. Let each blob's appearance change, but only along a small number of fixed directions learned per subject, which we call appearance blendshapes. Their mixing weights are a smooth function of facial muscle activations (Action Units, AUs) predicted from the text prompt and audio. Appearance can then express emotion-specific detail. Its frame-to-frame change is mathematically bounded by how fast the muscle signal changes. The extra rendering cost is one small weighted sum per blob.
Core Claim
The tension is stated clearly in the literature. TalkingGaussian freezes each blob's color and opacity and predicts only position, scale and rotation offsets. It argues that "appearance modification" is what causes ghosting and flicker in neural-field talking heads, and it reaches 108 FPS. EmoDiffTalk maps CLIP text embeddings to AU vectors and then runs an AU-conditioned diffusion model over Gaussian offsets. It adds a dedicated opacity network, because text-driven wrinkles need opacity change. It reports 25.78 PSNR on EmoTalk3D. It denoises every frame independently, and no frame rate is reported on Lacuna for it.
Our claim is that most of what matters in emotional appearance change is low-rank and AU-locked. If that holds, a structured parameterization can recover EmoDiffTalk-level expressiveness at TalkingGaussian-level stability and speed. For blob $i$ at frame $t$, opacity (and, analogously, color) is
$$o_i(t) = o_i^{0} + \sum_{k=1}^{K} a_k(t)\,\Delta o_{ik}, \qquad a(t) = g(\mathrm{AU}(t)).$$
Here $o_i^{0}$ is the frozen canonical opacity, $\Delta o_{ik}$ are $K$ learned per-blob residuals (with $K$ around 8 to 16), and $g$ is a small network mapping the AU vector $\mathrm{AU}(t)$ to mixing weights $a(t)$. By the triangle inequality,
$$|o_i(t+1)-o_i(t)| \le \|\Delta o_i\|\,\|a(t+1)-a(t)\|,$$
where $\|\Delta o_i\|$ is the norm of blob $i$'s residual vector. Appearance flicker is therefore controlled by the smoothness of a single $K$-dimensional trajectory rather than by millions of independent per-blob predictions.
The bound also changes how the multi-term loss is built. Composite losses for Gaussian avatars normally balance $L_1$, D-SSIM, perceptual and geometric terms. Adding a temporal term in pixel space usually means optical-flow warping in the style of TSAM, which aligns adjacent-frame features by flow and masks unreliable regions with cycle-consistency checks. That approach breaks exactly where mouths and brows move fast. We instead place the temporal penalty on $a(t)$ and $\mathrm{AU}(t)$. These are low-dimensional, need no flow, and give a weight that is easy to tune because its effect on pixels is bounded.
Research Program
The system has three parts. Each is deliberately borrowed except the middle one.
The first part is a feedforward text-and-audio-to-AU controller, in the spirit of InstructAvatar. InstructAvatar maps natural-language instructions to AU and motion sequences using GPT-4V-annotated data. Our controller adds per-word intensity modulation like the Fusion Intensity Modeling of MEDTalk, which uses Whisper and RoBERTa to make emotion intensity vary word by word. It is a single forward pass with a causal temporal convolution, so prompts can change mid-utterance without the window seams that CapTalk reports for its autoregressive window conditioning.
The second part, which is the contribution, is a per-subject Gaussian head. Geometry is bound to a FLAME mesh as in FlashAvatar, which samples blobs uniformly over the mesh surface and reaches 300 FPS at 512×512. The mouth is decomposed into face and oral-cavity parts as in TalkingGaussian. Geometric deformation is driven by FLAME parameters plus AUs. Appearance follows the blendshape equation above.
The third part is the loss. It combines photometric $L_1$, D-SSIM and LPIPS. It adds a binding regularizer like the $\mathcal{L}_{\text{angle}}$ of GeoAvatar, which keeps blobs on their mesh faces. It adds an AU-consistency term, scored by an off-the-shelf AU detector run on rendered frames. Its temporal terms are a second-difference penalty on $a(t)$, a sparsity prior that confines each $\Delta o_{ik}$ to a facial region, and an orthogonality prior across the residuals.
The analysis component tests the low-rank hypothesis directly. We first fit an unconstrained per-frame opacity and color residual, as an oracle, on MEAD subjects across all 8 emotions and 3 intensities. We then measure how much of its variance the top $K$ principal components capture, and how well $\mathrm{AU}(t)$ predicts their coefficients. If $K\le 16$ captures most of the wrinkle-region residual, the design is justified. If it does not, the result is still informative: it would show that emotional appearance in Gaussian avatars is genuinely high-dimensional, and the persistence argument has a real cost.
Baselines
The naive reference is TalkingGaussian-style frozen appearance with AU conditioning added to its deformation network. This is our model with $K=0$. It is fully stable and fast, and it shows how much expressiveness geometry alone delivers. Matching it on flicker while beating it on wrinkle-region LPIPS and AU F1 under strong-intensity prompts is the minimum bar.
The strongest existing method on this exact question is EmoDiffTalk. It is the only system in the file that renders text-prompted emotional detail on 3D Gaussians. Its reported number is PSNR 25.78 on EmoTalk3D, from EmoDiffTalk. We will evaluate on EmoTalk3D with the same subjects, splits and resolution. If code is unavailable, we will reimplement its AU-conditioned diffusion and opacity network on our shared geometry backbone, so that only the appearance mechanism differs. Beating it means matching PSNR within 0.5 dB while showing lower temporal high-frequency energy in wrinkle regions and at least 100 FPS, against its measured frame rate on the same GPU.
The key ablation replaces the low-rank residual with a full per-frame MLP that predicts each blob's opacity and color offset from AUs. This is the unconstrained "OPCNet-style" variant. We will also apply an exponential moving average to its outputs, which gives the same model temporal smoothing after the fact. These two variants isolate whether the low-rank structure itself, rather than the AU conditioning or the smoothing, produces the stability.
The pipeline baseline chains a text-to-FLAME generator into FlashAvatar. We will use either CapTalk, which reports lip vertex error 6.44 on CapTalkingHead, or our controller's output converted to FLAME. All per-subject avatars get the same training frames, the same 30-minute-class optimization budget on one RTX 3090-class GPU, and the same hyperparameter search size.
Why not just…?
Why not just generate FLAME codes from text and render them with FlashAvatar? This is the real threat. FlashAvatar already conditions small blob offsets on expression codes. It may therefore capture wrinkles through geometry alone at 300 FPS. Against it, FlashAvatar itself notes that rendering quality is bounded by FLAME tracking and that fixed-topology mesh models struggle with fine wrinkles. FLAME expression codes were also not designed around AU-level brow and forehead actions. The experiment settles it directly. If the pipeline matches our wrinkle-region LPIPS and AU F1 on MEAD's strong-intensity clips, the appearance blendshapes are unnecessary, and we would report that.
Why not just distill EmoDiffTalk's diffusion into one step? Distillation addresses speed, but it does not address stability. A one-step per-frame predictor of per-blob opacity is still an independent high-dimensional output at every frame, which is exactly our full-MLP ablation. If that ablation flickers in wrinkle regions and the low-rank model does not, distillation alone is insufficient.
Why not just smooth the outputs over time? Temporal smoothing of per-blob opacity lags fast prompt changes and blurs genuine transient detail such as a quick brow flash. The exponential-moving-average ablation measures this tradeoff explicitly, using onset sharpness at prompt switches against flicker.
Why not a 2D diffusion portrait model? EmoCAST reaches 83.60% emotion accuracy on MEAD. It is per-frame latent diffusion, and the file documents identity drift under intense prompts. We include it only as an expressiveness reference, not as a real-time competitor.
Proposed Experiments
We train per-subject avatars for MEAD actors, using the frontal view plus held-out side views. MEAD is described in the MEAD dataset paper, which reports 86.26% automated emotion classification accuracy. Prompts are produced by converting emotion and intensity labels plus detected AUs into free-text instructions with a vision-language model, following InstructAvatar's annotation pipeline. EmoTalk3D provides the head-to-head with EmoDiffTalk. HDTF gives a neutral-speech sanity check against GaussianTalker, which reports 98 FPS and 121 FPS for its light variant.
Metrics fall into four groups. Image quality uses PSNR, SSIM and LPIPS, both globally and inside forehead and nasolabial masks. Lip sync and expression use Sync-score and landmark distance. Emotion control uses emotion accuracy and AU F1. Stability uses the ratio of rendered to ground-truth temporal high-frequency energy in wrinkle masks, together with VFID. Speed is FPS at 512×512.
The decisive checks are four. First, the PCA study must show that low $K$ captures the oracle residual. Second, the $K$ sweep (0, 4, 8, 16, full) should trace a curve in which expressiveness saturates early and flicker rises only for the full variant. Third, under mid-utterance prompt switches, our causal controller should show no seam artifacts, measured as jumps in $\|a(t+1)-a(t)\|$ and in pixels at the switch. Fourth, the FLAME-plus-FlashAvatar pipeline must fall short on strong-intensity wrinkle metrics. If that fourth check fails, the proposal reduces to a better text-to-FLAME controller, and we would say so.
References
- TalkingGaussian: Structure-Persistent 3D Talking Head Synthesis via Gaussian Splatting
- EmoDiffTalk: Emotion-aware Diffusion for Editable 3D Gaussian Talking Head
- Progressive Temporal Feature Alignment Network for Video Inpainting
- InstructAvatar: Text-Guided Emotion and Motion Control for Avatar Generation
- MEDTalk: Multimodal Controlled 3D Facial Animation with Dynamic Emotions by Disentangled Embedding
- CapTalk: Text-Guided Stylization and Speech-Driven 3D Head Animation
- FlashAvatar: High-Fidelity Head Avatar with Efficient Gaussian Embedding
- GeoAvatar: Adaptive Geometrical Gaussian Splatting for 3D Head Avatar
- EmoCAST: Emotional Talking Portrait via Emotive Text Description
- MEAD: A Large-Scale Audio-Visual Dataset for Emotional Talking-Face Generation