Systems that animate a 3D avatar from speech must decide how much the face model and the body model should talk to each other. If the body informs the face, a stray, random arm movement can disturb the mouth and ruin lip sync. If the body cannot inform the face, the avatar loses natural pairings such as a shrug accompanying a lopsided smirk. We propose to settle this with a selection rule borrowed from crystal physics. In a crystal, two electronic states that respond differently to a symmetry operation cannot mix, because the term that would couple them is forced to be exactly zero. Speech itself has no left or right. So under a left–right mirror, lip-sync motion (jaw opening, lip closure, pucker) is mirror-symmetric, while most of what makes gestures unpredictable (which hand moves, a lean to one side) is mirror-antisymmetric. If face–body coupling is built so that mirror-symmetric face channels can only read mirror-symmetric body features, lateral gesture noise cannot reach the lips, and the antisymmetric face channels stay free to pick up the shrug-and-smirk correlations.
The physics supplies a second, testable idea. A protected degeneracy, meaning two energy levels forced by symmetry to coincide, splits in proportion to the perturbation that breaks the symmetry. The cleanest demonstration is Strain-induced nonsymmorphic symmetry breaking in SrIrO3, where a 0.4° shear of the crystal breaks one glide symmetry while keeping a mirror, and opens a gap of about 5 meV. Real speakers break mirror symmetry too: some are right-handed and some have asymmetric faces. We therefore model each speaker's asymmetry as an explicit, learned symmetry-breaking parameter, and predict that leakage from lateral gestures into the lips grows linearly with that parameter and no faster. A glide-type operation, which combines a reflection with a fractional translation, generalizes the mirror. We test it as a secondary hypothesis in time rather than space: a mirror combined with a time shift, which may capture speakers who alternate hands from beat to beat.
Core Claim
The disagreement between DiffSHEG and AsynFusion is a partitioning problem. DiffSHEG feeds face information to the body generator in one direction only and argues that any feedback from body to face corrupts the lips. AsynFusion's ablations show that two-way cross-attention (its CoSync module) is needed to keep the two modalities balanced, and the adapter-based joint diffusion model likewise relies on two-way adapters to capture weak cross-modal correlations. Both camps are right about different subspaces of the face. We claim that the mirror-parity split separates the protected subspace from the free one better than a hand-drawn region mask such as "mouth versus upper face," for two reasons. Jaw and cheek channels co-articulate with speech but sit outside the mouth region. Asymmetric mouth motion, such as a smirk, is expressive and belongs to the gesture-coupled side.
Write each face or body channel vector $x$ in a basis of symmetric and antisymmetric combinations of left and right counterparts: $x = (x^{+}, x^{-})$ with $Mx = (x^{+}, -x^{-})$, where $M$ is the mirror. Let $c$ be the speaker-specific asymmetry parameter and $\epsilon = \lVert c \rVert$ its size. If the network that predicts the symmetric face channels $f^{+}$ takes as input only the audio, the symmetric body features $b^{+}$, and $c$, then the response of those channels to a lateral body perturbation $\delta b^{-}$ obeys
$$\Big\lVert \frac{\partial f^{+}}{\partial b^{-}} \Big\rVert = 0 \ \text{at}\ \epsilon = 0, \qquad \Big\lVert \frac{\partial f^{+}}{\partial b^{-}} \Big\rVert \le C\,\epsilon \ \text{for small}\ \epsilon .$$
The linear scaling comes from a first-order expansion: the only route from $b^{-}$ to $f^{+}$ is through the term that breaks the symmetry, exactly as a gap opens linearly with strain.
We state the limits of this protection plainly. A network that is merely mirror-equivariant still lets symmetric outputs depend on even functions of antisymmetric inputs, such as their magnitude. Equivariance alone would therefore only make the lips blind to which side a gesture is on. The guarantee needs the triangular routing above, in which the symmetric face sector never reads the antisymmetric body sector except through $c$. The symmetry tells us how to partition the channels; the architecture enforces the partition. Symmetric body noise, for example both arms rising together, is not protected, and we measure how much of the gesture variance it carries.
Research Program
We build on a two-branch diffusion transformer of the AsynFusion type and keep its asynchronous few-step sampling, so that speed comparisons stay fair.
The first step is constructing the mirror basis. For SMPL-X body joints and FLAME-style blendshapes, the standard body and face rigs used in BEAT and SHOW, left and right counterparts are known, so the symmetric and antisymmetric combinations are fixed linear maps. Self-symmetric channels such as jaw open and spine need sign conventions, which we verify by mirroring captured frames and checking that the reflection maps them onto themselves.
The second step is the coupling layer. Cross-attention from body to face is split by sector. Face-symmetric queries attend to symmetric body keys and values only. Face-antisymmetric queries attend to everything. Face-to-body flow is left unrestricted, as in AsynFusion.
The third step is the symmetry-breaking term. Each speaker receives a low-dimensional antisymmetric vector $c$, and $c$ is the only path between sectors. Training adds an L1 penalty on $c$, so the model buys asymmetry only where the data require it, much as the crystal admits only the strain the substrate imposes.
The fourth step is the glide extension. Here body tokens are allowed to couple across sectors only under the combined operation "mirror plus a shift of one beat," with beats taken from audio onsets. This tests whether alternating-hand gestures are better described by a time-shifted symmetry than by a plain one. We give this hypothesis a lower prior and report it separately. If it fails, the core claim does not depend on it.
Baselines
The naive reference is a fully decoupled pair of face and body generators that share only the audio encoder. It is the lower bound on cross-modal harmony and the upper bound on lip isolation. Every coupled model should match its lip accuracy while beating its joint realism.
The strongest existing method is AsynFusion. Its two-way CoSync attention and feature buffer are the most direct competitor on whole-body coupling, and it reports FMD 312.46 and FGD 421.58 on BEAT (lower is better for both Fréchet distances). We will retrain it ourselves on the same BEAT and SHOW splits, with the same feature extractors, sampling step counts and tuning budget, because the BEAT paper's CaMN result of FGD 123.7 (BEAT) is clearly on a different scale and cannot be compared directly with AsynFusion's number.
DiffSHEG is the one-way, face-to-body baseline. It is the strongest competitor on lip protection specifically. We found no reported FGD, FMD or facial-error numbers for DiffSHEG on Lacuna, so it is retrained under the identical protocol.
The ablation that isolates our key ingredient is a region-masked two-way model, identical to ours except that the protected set is "mouth and jaw blendshapes" rather than the mirror-symmetric sector. A second ablation keeps the parity routing but removes the asymmetry parameter $c$, which tests whether controlled symmetry breaking matters. A third trains AsynFusion with random mirror augmentation, which encourages the symmetry statistically instead of building it in.
Beating AsynFusion means matching its FMD and FGD within seed variance while improving lip accuracy and cutting measured leakage. Beating the region mask means better joint expressiveness, measured by the asymmetric face–gesture co-occurrence score defined below, at equal or better lip accuracy.
Why not just…?
Why not just mask body-to-face attention to the non-mouth channels? This is the real threat, since a mask in the right basis is what our method amounts to. The question is whether region or parity is the right partition. A region mask forbids exactly the smirk-with-shrug coupling that the adapter-based joint model treats as essential, while still exposing jaw-adjacent channels if they are not in the mask. The region-mask ablation decides this directly. If it matches parity routing on both lip accuracy and the co-occurrence score, we will report that the physics framing adds nothing beyond a good mask.
Why not just keep two-way cross-attention and trust training, as AsynFusion does? AsynFusion gives no guarantee about where gesture randomness flows, and DiffSHEG's argument for one-way flow rests on observing exactly that corruption. Our noise-injection test below measures whether CoSync leaks lateral gesture noise into the lips. If the leakage is negligible, the protection is unnecessary on these datasets, and we will say so.
Why not just augment with mirrored data? Augmentation makes the distribution symmetric but leaves cross-sector paths inside the network, so the lips can still react to which side a gesture is on. It also cannot represent a speaker's genuine handedness except as noise. The augmented AsynFusion baseline, together with the predicted linear leakage-versus-$\epsilon$ curve, separates "symmetric on average" from "symmetric by construction."
Why not just rely on phoneme keyframe anchoring? KMTalk locks lips at forced-alignment boundaries and reports a 30% lower lip-vertex error on VOCASET relative to SelfTalk. However, it handles the face only and breaks under noisy speech or alignment errors. Our method needs no aligner at inference time. We use the Montreal Forced Aligner only to score evaluations.
Proposed Experiments
All models are trained on BEAT, which has dense face blendshapes, body and finger motion, and 30 speakers, which is enough to estimate per-speaker asymmetry. SHOW, with four speakers reconstructed from video, serves as the noisier transfer set.
The decisive test is leakage. At inference we add Gaussian noise of fixed energy to the antisymmetric body latents, or swap the left and right arm trajectories, and measure the change in mouth channels at bilabial closures located by the Montreal Forced Aligner on the clean audio. The parity model should show leakage near zero for speakers with small $\epsilon$ and roughly linear growth across speakers ranked by learned $\epsilon$. AsynFusion and the augmented baseline should show leakage that does not track $\epsilon$.
Quality is measured by FMD and FGD on BEAT, and by lip-closure accuracy together with a lip-vertex-style blendshape error. Expressiveness is measured by a co-occurrence score, defined as the mutual information between antisymmetric face channels and antisymmetric body channels in generated sequences compared with held-out real ones. The decoupled and region-masked models should lose on this score, while the parity model should not.
The glide hypothesis is tested on BEAT segments with alternating-hand beat gestures, identified by the sign of the lateral velocity of the wrists. It passes only if the time-shifted variant improves FGD on those segments without hurting the rest.
The result that would sink the core claim is region masking matching parity routing on every axis, or leakage under AsynFusion already being as low as under our model. Either outcome would itself answer the open DiffSHEG-versus-AsynFusion question.
References
- Strain-induced nonsymmorphic symmetry breaking and removal of Dirac semimetallic nodal line in an orthoperovskite iridate
- DiffSHEG: A Diffusion-Based Approach for Real-Time Speech-driven Holistic 3D Expression and Gesture Generation
- AsynFusion: Towards Asynchronous Latent Consistency Models for Decoupled Whole-Body Audio-Driven Avatars
- Joint Co-Speech Gesture and Expressive Talking Face Generation Using Diffusion with Adapters
- BEAT: A Large-Scale Semantic and Emotional Multi-modal Dataset for Conversational Gestures Synthesis
- KMTalk: Speech-Driven 3D Facial Animation with Key Motion Embedding