Signed Geometric Phase as a Diagnostic and Cure for Mean-Pose Collapse in Gesture Synthesis

When a neural network learns to turn speech into body motion by minimizing squared error, it learns the average of all the motions that could accompany that speech. If a speaker sometimes circles a hand clockwise and sometimes counterclockwise on the same phrase, the average is a hand that barely moves. This "regression to the mean" is the most persistent failure in speech-driven animation. We propose to measure it with a quantity borrowed from condensed-matter physics, the geometric (Berry) phase. For a limb, the geometric phase is the signed solid angle its pointing direction sweeps out over a short window: positive for one rotation sense, negative for the other. Opposite rotation senses cancel to roughly zero under averaging.

The proposal has two parts. First, the signed geometric phase gives a direct, rotation-invariant test for collapse: the distribution of phases in generated motion shrinks toward zero exactly when the model averages across rotation senses. Second, a generator can sample a coarse discrete phase label per window and regress the remaining continuous detail. This should recover much of the vividness that full motion tokenizers or diffusion models buy, at the cost of a small classifier. We borrow one specific tool from physics: the discrete Wilson-loop product used to compute topological invariants of materials, which gives the phase without choosing a reference frame or projection plane.

Core Claim

Mean-pose collapse in co-speech gesture is, to a measurable degree, a collapse of oriented quantities, meaning quantities with a sign that averaging can cancel. The signed geometric phase of each limb's direction is the cleanest such quantity, and it can be computed robustly from sampled motion. We claim three things, each of which can be falsified.

The first claim is that models trained with squared or absolute error show a geometric-phase distribution that is much narrower than real data. They also show a sign balance near zero within windows where the data are bimodal.

The second claim is that this narrowing is not explained by low kinetic energy alone.

The third claim concerns a split model. A model that samples a three-way phase label per window by classification, then regresses motion conditioned on it, should close a large part of the gap to discrete-token and diffusion generators on standard gesture benchmarks.

The physics supplies the computation. Represent the unit direction $n_t$ of a limb segment (for example forearm or wrist relative to shoulder) at frame $t$ as a two-component spin state $|\psi_t\rangle$ whose expected spin points along $n_t$. Over a window of $T$ frames, closed by returning from $n_T$ to $n_1$ along the shortest great-circle arc, the discrete phase is

$$\gamma = -\arg \prod_{t=1}^{T} \langle \psi_t | \psi_{t+1} \rangle, \qquad \psi_{T+1} \equiv \psi_1,$$

and it equals $-\Omega/2$ modulo $2\pi$, where $\Omega$ is the signed solid angle enclosed by the limb's path on the sphere of directions. Each $|\psi_t\rangle$ carries an arbitrary phase factor, but every factor appears once as a bra and once as a ket in the product, so it cancels. This is the same property that lets the Wilson-loop method sidestep the gauge problem in electronic structure. In that setting, Sheng et al. use products of overlaps around closed paths to extract the $\mathbb{Z}_2$ index of thallium nitride without constructing a smooth gauge. For motion, the practical consequences are three. The phase needs no hand-chosen projection plane. It is unchanged by a global rotation of the body. It does not break on angle wrapping or the double-cover sign ambiguity of quaternions.

Where This Sits in Prior Work

Several papers motivate the proposal by documenting the failure. Audio2Gestures shows that lower $L_1/L_2$ error goes with less realistic, less diverse gesture, and responds with a split latent code and a loss that only penalizes errors beyond a margin. Kucherenko et al. describe regression output that drifts toward a static center. ProsodyTalker and Theobald et al. report the same dampening in upper-face motion. Sign Stitching explains why it matters most in sign language, where movement direction carries meaning. None of these papers names which statistics of the motion are destroyed. Our diagnostic is meant to say so precisely for rotation sense.

Two lines of work supply the competing cures. Discrete tokenization, as in PoseGPT and RACER, sidesteps averaging by having the model pick one entry from a codebook built by Euclidean clustering. G2P-DDM does the same with discrete diffusion. Stochastic and adversarial generation is the other line. It includes DiffSHEG, which uses diffusion, and the trimodal adversarial model of Yoon et al.. Our proposal is narrower than both lines. It discretizes one interpretable, oriented quantity and leaves everything else to plain regression. The point is to test whether that single cut carries much of the benefit. Taşyürek et al. split motion by articulator so that one body part's variance does not wash out another's. We instead split each articulator's motion by rotation sense, and the two splits can be combined.

The closest topological work is Topological Autoencoders, which match the persistent homology (counts of components and loops) of data and latent spaces. Persistent homology does not record orientation: a clockwise loop and a counterclockwise loop give the same signature. That is exactly the information averaging destroys, so a persistence-based loss is the wrong tool for this failure. The geometric-mechanics gait work of this RSS paper computes loop integrals of a connection over cyclic shape changes to predict a robot's net displacement. That supports our premise that loop holonomies summarize cyclic motion compactly, though that work involves no learning or conditioning. Finally, the symmetry-indicator approach of the Catalogue of topological electronic materials is the conceptual template for the split model: predict a coarse discrete label that cannot be meaningfully averaged, then let continuous detail vary within it.

Research Program

The first stage builds the measurement. For each of the 4–6 major upper-limb segments, we compute $\gamma$ over sliding windows of roughly 0.5–1.5 seconds. Window length will be swept, because geometric phase depends on scale. This gives a per-clip "phase spectrum": the joint histogram of $\gamma$ across limbs and window lengths. We check numerical stability against frame rate by downsampling motion-capture data and confirming that $\gamma$ converges. Convergence is expected because the discrete overlap product approximates the continuous holonomy as the steps shrink. We also check stability against the closing-geodesic choice by comparing it with longer overlapping windows. For non-Abelian structure, such as the twist of the full bone frame, we keep an optional extension that uses the eigenphases of the product of $SU(2)$ frame overlaps. We pursue it only if the Abelian phase proves insufficient.

The second stage is the diagnostic study. We train an $L_2$ regressor, an $L_1$ regressor, and the published-style baselines on the same data. For each model we compare its phase spectrum with the real data's, measuring the Wasserstein distance between the $\gamma$ histograms and the sign-balance statistic conditioned on audio-matched windows. To separate rotation sense from mere slowness, we stratify windows by mean limb speed. We then ask whether the collapse in $|\gamma|$ and in sign balance persists at matched kinetic energy.

The third stage is the split generator. We assign each training window a phase label for each limb: strongly negative, near zero, or strongly positive, with thresholds set by data quantiles. An audio and text encoder predicts these labels with cross-entropy, and at inference the labels are sampled rather than taken by argmax. A continuous decoder produces joint rotations conditioned on the sampled labels under $L_2$ loss, plus a light penalty that keeps the generated $\gamma$ inside the sampled label's interval. Speech probably does not determine rotation sense, so the classifier is expected to output spread-out probabilities. Sampling from them preserves the multimodality that a regression head would erase. That is the mechanism the proposal relies on.

Baselines

The naive reference point is a deterministic $L_2$ regressor with the same encoder and decoder as our split model but no phase labels. It is the collapsed model the diagnostic is built to expose. To beat it, the split model must improve Fréchet Gesture Distance (FGD, a distance between feature distributions of real and generated motion) and phase-spectrum distance by a margin larger than the spread across random seeds.

The strongest existing competitor on the Trinity dataset is RACER, which uses tokens from a vector-quantized autoencoder (VQ-VAE) and a policy trained by offline reinforcement learning against contrastive audio–motion rewards. Sun et al. report FGD 13.44 against 17.29 for their best baseline, with 89.58% of matched beats. On the TED dataset the strongest reported system is the trimodal adversarial model of Yoon et al., with FGD 3.729. Removing its adversarial term raises that to 9.712, which tells us how much of the gain comes from fighting collapse. For DiffSHEG we found no reported result on Lacuna on these benchmarks, so we will retrain it ourselves. Audio2Gestures reports a Trinity diversity score of 6.32, against 2.46 for Speech2Gesture.

All retrained baselines share our data splits, skeleton representation, FGD feature extractor, GPU-hour budget, and number of hyperparameter trials. We do not expect a three-label split to beat RACER outright. It counts as a success if it recovers at least half of the FGD gap between the $L_2$ regressor and RACER, and matches RACER on phase-spectrum distance, with a fraction of the parameters and no reinforcement learning.

Three ablations isolate the key ingredient, and each replaces our phase labels while keeping vocabulary size and decoder fixed. The first uses labels from the signed area of the wrist path projected onto a fixed frontal plane. The second uses only unsigned $|\gamma|$. The third uses a VQ-VAE codebook of matched size. An oracle that conditions on ground-truth labels gives the ceiling on what the split can achieve.

Why not just…?

Why not just use diffusion or VQ tokens, which already escape the mean? They do, and on raw FGD they may well win. Our claim is not that the geometric phase beats them. The claim is that it identifies which information they restore, and that a three-label variable recovers much of it cheaply and interpretably. The matched-vocabulary VQ ablation settles this. If clustered codes of the same size match our labels, then orientation is not special, and we will report the geometric phase purely as a diagnostic.

Why not just measure velocity, jerk, or a diversity score? Energy statistics cannot tell a clockwise circle from a counterclockwise one. A diversity score across samples says nothing about whether each sample matches the data's distribution over rotation sense. The speed-stratified analysis in the diagnostic study is the test. If collapse in $\gamma$ disappears once kinetic energy is matched, this alternative wins and the oriented framing is unnecessary.

Why not just compute the signed area in a projected plane, with no physics machinery? This is the real threat, because the two quantities agree whenever gestures stay near one plane. The overlap product matters when gestures change plane, when the body turns, or when angles wrap, since a fixed projection then mixes rotation senses. The planar-label ablation decides it. On BEAT's more varied full-body clips we expect the solid-angle labels to give a clearly lower phase-spectrum distance and FGD. If they do not, we will adopt the simpler planar statistic and say so.

Why not a persistent-homology loss like Topological Autoencoders? Persistent homology ignores orientation, which is the property averaging cancels. The unsigned-$|\gamma|$ ablation is a close stand-in. If it performs as well as signed labels, the sign carries nothing extra and the argument against persistence-style losses fails.

Proposed Experiments

The experiments use the Trinity Gesture Dataset, the expanded TED Gesture Dataset, and BEAT. BEAT is included because its emotional range and full-body capture should produce more out-of-plane, sign-varying gestures. HumanAct12 serves as a control where conditioning is a discrete action label and collapse should be weaker. The metrics are FGD, beat matching, Audio2Gestures-style diversity, and our phase-spectrum Wasserstein distance. We will also run a small two-alternative forced-choice user study on clip pairs that differ mainly in rotation sense.

The first decisive check is whether $L_2$ models show the predicted shrinkage of $\gamma$ toward zero at matched speed; if not, the premise fails. The second is whether the oracle-label model clearly beats the plain regressor, which tells us whether rotation sense is worth predicting at all. The third is whether sampled signed labels beat unsigned, planar, and clustered labels of equal size, which tells us whether the gauge-invariant computation earns its place. The fourth checks that the classifier's sign entropy is high on bimodal windows, confirming that the gain comes from sampling rather than from audio actually determining rotation sense. Finally, as an extension, we will test the same labels on sign-language production, where rotation sense is part of what a sign means.