Style Lives in the Tails: Restoring Higher-Order Motion Statistics Lost in Latent Motion Diffusion

Text-to-motion generators produce believable walking or jumping, but the manner of the movement, such as a stomp, a swagger or a limp, tends to fade into a generic average. We argue that much of this fading comes from a specific place: the autoencoder that compresses motion before generation. A latent diffusion model is a generator that learns to turn noise into samples inside a compressed code produced by an autoencoder. In such models, the temporal compression acts like a low-pass filter. It keeps the smooth bulk of a movement and discards sudden accelerations, asymmetric decelerations and foot-strike bursts. Prior work on style transfer finds that these are the statistics that tell styles apart. Two such statistics matter here: skewness, which measures how lopsided a distribution is, and kurtosis, which measures how peaked and burst-prone it is. If this diagnosis is right, no amount of better text conditioning can recover a style that the decoder cannot represent.

We propose to test that diagnosis directly and then repair it cheaply. The repair adds a small, explicit side channel of windowed higher-order statistics of joint acceleration. This channel goes around the motion autoencoder, is generated jointly with the ordinary latent, and is the target to which text style prompts are aligned. An adversarial dataset discriminator then removes capture-specific differences from the content latent only. This is deliberate, because applying invariance pressure to everything would erase the style we are trying to keep. The note's central bet is that a few dozen interpretable numbers per time window recover what a much larger learned latent would. That bet is falsifiable with a capacity-matched control.

Core Claim

The claim has three linked parts. First, the compressed latent of a standard motion autoencoder, such as the transformer motion VAE (variational autoencoder) of MLD, measurably loses style. The loss should concentrate in third- and fourth-order statistics of joint velocity and acceleration, not in means and variances. Second, an explicit side channel of these moments restores style recognizability at lower cost than enlarging the latent by the same number of dimensions. Third, because this channel is low-dimensional and carries the style signal directly, short text style labels align to it better than to the full latent. A dataset-adversarial loss improves cross-dataset content transfer only when it is confined to the content latent.

The motivating evidence comes from AStF. That paper matches mean, variance, skewness and kurtosis in skeletal feature space, and shows that styles which overlap under mean and variance become separable once the higher moments are added. AStF works without a generative latent and without text, so it never asks whether a latent bottleneck destroys these moments. Three motion stylization systems do run in a latent space, but none of them targets higher-order statistics explicitly. StyleMotif injects mean/variance-normalized style features into MLD and aligns text through an ImageBind projection. MCM-LDM strips style from content with instance normalization, which removes only the first two moments, and takes style from MotionCLIP. LEAD realigns the VAE latent to CLIP and learns pseudo-words from a few clips. VQ-Style is the closest structural relative: it puts style in residual codebooks of a quantized autoencoder. However, it is motion-to-motion only, its style codes are learned rather than tied to physical statistics, and it accepts no text prompt.

Research Program

The work begins with a diagnostic, because the rest of the program depends on it. We take the pretrained MLD motion VAE and encode and decode every clip of 100STYLE, Xia and BFA. We then compare reconstructions with the originals in two ways. The first is the accuracy of a fixed, separately trained style classifier. The second is the per-joint error in windowed mean, variance, skewness and kurtosis of velocity and acceleration. We repeat this across VAE variants with different latent sizes and temporal compression rates. This produces a "style ceiling" curve: the best style accuracy any downstream generator could reach given that decoder. We found no reported result on Lacuna for style accuracy of motion VAE reconstructions, so this measurement is new information either way.

The repair attaches a moment channel to the latent. For each clip $x$, we compute $m = (m_1, \dots, m_W)$, where $m_w$ holds the standardized third and fourth moments of acceleration for a small set of end-effector and root joints over window $w$ (about 16 frames). Short windows make sample kurtosis noisy. We will therefore compare raw moments with robust variants, L-moments (rank-based analogues of skewness and kurtosis), and pick the variant with lower variance across repeated takes of the same style. The decoder is conditioned on both the VAE latent $z$ and $m$. It is trained with

$$\mathcal{L} = \mathcal{L}_{\text{rec}}(x, \hat{x}) + \lambda_m \lVert m(\hat{x}) - m \rVert_1 - \lambda_d \, \mathcal{L}_{\text{dom}}(D(z)),$$

where $\hat{x}$ is the decoded motion and $m(\hat{x})$ its recomputed moments. $D$ is a discriminator that predicts which dataset a clip came from (HumanML3D, 100STYLE, CMU Mocap). Its loss $\mathcal{L}_{\text{dom}}$ reaches $z$ through a gradient reversal layer, which flips the sign of gradients flowing back to the encoder. $\lambda_m$ and $\lambda_d$ are weights. The discriminator never sees $m$. This placement answers the tension between the two literatures. Domain adaptation work treats style as nuisance and erases it, as in FAN-Net, which uses gradient reversal to strip site-specific style from medical images. In motion stylization, style is the payload. Our design lets the adversary act only where style is meant to be absent.

The diffusion model then denoises the concatenation of $z$ and $m$, conditioned on content text and a style prompt. Style prompts follow LEAD's projector recipe, but the projector aligns them with a contrastive loss to pooled moment trajectories rather than to the full latent. Following StyleMotif's finding, we use concise 100STYLE labels, since that paper reports short labels align better than detailed descriptions. Style transfer from a reference clip needs no encoder at all, because $m$ can be computed directly from the reference.

Baselines

The naive reference point is unmodified MLD with style text appended to the content prompt. It shows what happens when the bottleneck is left alone, and MLD reports a HumanML3D FID (Fréchet distance between feature distributions of real and generated motion) of 0.473.

The strongest competitors on text- or motion-guided stylization inside a latent diffusion model are MCM-LDM and StyleMotif. MCM-LDM reports Fréchet Motion Distance 27.69 and Style Recognition Accuracy 58.00% on HumanML3D. StyleMotif reports text-guided Style Recognition Accuracy of 56.71%. We will retrain both from released configurations, or reimplement them, on the same HumanML3D plus 100STYLE split, with the same diffusion backbone size, training steps and a matched hyperparameter search budget. We will score everything with one shared style classifier, because the reported numbers come from different evaluators. Beating them means higher style recognition accuracy at equal or better FID and content consistency under our shared evaluator. It also means a smaller gap in skewness and kurtosis to held-out real clips of the target style.

For pure motion-to-motion transfer, the strongest baselines are VQ-Style and AStF. VQ-Style reports 83.20% style preservation and 68.95% zero-shot accuracy on 100STYLE. AStF reports Style FID 0.157 and style accuracy 93.0% on Xia. Our method should at least match VQ-Style's zero-shot number on unseen 100STYLE styles. That comparison tests whether explicit moments generalize better than learned residual codes.

Two ablations isolate the key ingredient. The capacity-matched control removes $m$ and instead widens $z$ by exactly the same number of dimensions, trained with the same loss minus the moment term. The adversary-placement ablation applies the gradient reversal to $[z, m]$ jointly instead of to $z$ alone.

Why not just…?

Why not just make the VAE latent bigger or compress time less? This is the most serious threat. If a wider latent recovers style, the moment channel is unnecessary engineering. The capacity-matched control settles it directly. If the widened latent closes the style ceiling as well as the moment channel does, we will report that the bottleneck is a capacity problem rather than a statistics problem, and the alignment claim becomes the main result. We expect it not to close the gap at equal size. A reconstruction loss weighted by squared error is dominated by low-frequency energy, so extra capacity is spent there first. LEAD's account of a trade-off between latent quality and text alignment also suggests that larger latents cost something downstream.

Why not just use mean/variance style modulation (AdaIN) in the latent, as StyleMotif does? AStF's visualizations show that styles overlapping under mean and variance separate once skewness and kurtosis are included. A latent that has already lost those moments cannot be fixed by modulating its first two. The StyleMotif baseline, together with a moment-gap analysis split by order, will show whether its residual error lives in the third and fourth moments.

Why not just use residual quantized codes, as VQ-Style does? Residual codebooks capture fine detail, but nothing ties them to physically meaningful statistics. They are also discrete, which can quantize away continuous phase detail, and VQ-Style offers no text interface. If VQ-Style's residual codes, given a text projector, match our method on text-guided stylization, then the explicit-statistics hypothesis fails. We will run exactly that hybrid as an extra baseline.

Why not just skip the latent and diffuse in raw or frequency space? Raw-space diffusion avoids the bottleneck at roughly the latency cost MLD reports avoiding (about 24 s versus 0.2 s per batch). DiffusionPhase moves to periodic frequency parameters but has no style–content separation. Periodic parameterizations also suit cyclic gait better than isolated bursts such as a stomp. We include DiffusionPhase-style conditioning as a reference, not as a replacement.

Proposed Experiments

The experiments run on HumanML3D for content and text, 100STYLE for style labels and zero-shot splits, Xia and BFA for cross-dataset style evaluation, and CMU Mocap as unlabeled data for the adversarial objective.

The first decisive check is the style ceiling curve. If MLD-VAE reconstructions keep near-original style accuracy and moment statistics, the premise is wrong, and we stop and pivot to the alignment question alone.

The second check is the capacity-matched control described above. The proposal survives only if the moment channel beats a latent widened by the same number of dimensions on style accuracy and on skewness/kurtosis error, without worse FID.

The third check is text alignment. Using held-out 100STYLE labels, we test whether prompts aligned to the moment channel achieve higher style recognition accuracy than LEAD-style alignment to the full latent under the shared evaluator.

The fourth check concerns the adversary. Gradient reversal on $z$ alone should improve content metrics when content comes from CMU Mocap and style from 100STYLE. Applying it to $[z, m]$ should measurably lower style accuracy. That would be direct evidence of the negative transfer that the domain adaptation literature warns about when the shifted variable is the one the task cares about.

A small user study will compare stomp-, limp- and strut-type styles, where bursts matter most. It will check that gains in the statistics correspond to motion people perceive as more stylized and not merely jerkier. Foot-sliding rate will be reported throughout to catch that failure.