Models that generate human motion one step at a time, each step conditioned on what they already produced, are trained on clean recorded motion and then run on their own imperfect output. Small errors push later steps into territory the model never saw, and over long rollouts the character freezes, jitters, or slides. This proposal tests one fix. During ordinary parallel training, we perturb the model's internal activations for past frames, and we perturb them specifically in the directions the model's own rollouts are measured to drift. Clean anchors stay untouched: the text prompt, a distant target keyframe, a goal position for the hand. The model therefore learns to use attention over those anchors to pull itself back on course. We add nothing at inference time.
The opportunity matters because every current remedy has a cost. T2M-GPT replaces half the context tokens with random codes, which is crude and leaves jitter. COMET and Thermalizer bolt external correctors onto inference, which adds latency. Video representation regularization penalizes the geometry of hidden states but never shows the model what its errors look like, and it has never been tried on articulated motion. We claim that structured latent augmentation matched to self-generated drift, combined with clean long-range anchors, extends the length of stable and task-accurate motion rollouts beyond what these remedies achieve. The claim is decisive only if the structured version beats a plain isotropic-noise control and a compute-matched rollout-training baseline.
Where the prior work leaves the gap
Three strands motivate the idea. The first is Generating Human Motion from Textual Descriptions with Discrete Representations (T2M-GPT). It compresses motion into discrete codes with a vector-quantized autoencoder, which maps short motion chunks to entries in a learned codebook. A GPT-style transformer then predicts those codes from text. To fight exposure bias, T2M-GPT swaps a fraction τ=0.5 of ground-truth codes for random ones. That corruption is uniform over the codebook and bears no relation to the mistakes the model actually makes, and the authors still report leg and hand jitter.
The second strand is Mitigating Compounding Error via Video Representation Regularization. It finds that video rollouts fail after the effective rank of hidden states collapses. Effective rank is roughly the number of independent directions the representation actually uses. The paper counters this with a penalty that pushes hidden states toward an isotropic Gaussian. It also reports that more data or longer training can make collapse worse. We reuse its diagnostic, but we differ in mechanism: it constrains representations passively, whereas we feed the model realistic corrupted states and supervise recovery.
The third strand is two external correctors that monitor and repair rollouts at inference. Controllable Long-term Motion Generation with Extended Joint Targets (COMET) nudges poses toward a Gaussian mixture fitted to reference poses. Thermalizer runs a drift classifier plus reverse diffusion, and the classifier alone adds about 60% per-step overhead. Both treat drift as something to clean up after it happens. We try to make the base model not need the cleanup.
The augmentation technique itself comes from Implicit Semantic Data Augmentation for Deep Networks (ISDA). ISDA perturbs deep features along a class-conditional covariance, the directions in which examples of one class naturally vary. The same paper also constrains our design: it warns that covariance perturbations assume locally linear feature manifolds and corrupt states when scaled too aggressively. We change the source of the covariance. ISDA uses directions of semantic variation within a class. We use directions of self-generated error, and we apply the perturbation to context positions rather than to the classifier input.
Long-range context enters through the tasks. A Unified 3D Human Motion Synthesis Model via Conditional Variational Auto-Encoder* frames prediction, in-betweening and completion as one masked-sequence problem. It lets every frame attend to distant goal frames, and it credits that attention with preventing drift. T2LM avoids frame-by-frame drift by decoding with a strictly local receptive field, but the same restriction prevents it from reaching distant spatial targets. Our bet is that attention to clean distant anchors becomes an error-correction channel only if training forces the model to use it that way. In our setup, anchors are exactly the signal that augmentation leaves uncorrupted.
Core Claim
Consider a transformer that predicts motion codes. Let $h_{l,t}$ be its hidden state at layer $l$ for context position $t$ under teacher forcing, and let $\tilde h_{l,t}$ be the state at the same position when the preceding context is the model's own $k$-step rollout. Every $N$ training iterations we run short rollouts on a small held-out batch and estimate a low-rank drift covariance per layer and per rollout depth $k$:
$$\Sigma_{l,k} = \mathrm{Cov}\big[\tilde h_{l,t} - h_{l,t}\big].$$
During the main parallel training pass, we replace context states with $h_{l,t} + \delta$, where $\delta \sim \mathcal{N}(0, \lambda\, \Sigma_{l,k(t)})$. Here $k(t)$ is a depth sampled to mimic long-rollout conditions, and $\lambda$ is a scale ramped above 1 so that training covers error magnitudes larger than those currently observed. We leave anchor tokens (text, keyframes, joint targets) clean. We keep the loss as next-code cross-entropy against the clean targets.
The prediction is that this model sustains low effective-rank loss and low task error for far longer rollouts than models trained with random token corruption, isotropic hidden noise, or rank regularization. We further predict that the gain grows with the temporal distance to the nearest anchor, which is the signature of correction flowing through long-range attention.
Research Program
The first stage is diagnostic, and it is worth publishing on its own. We train a T2M-GPT-style model on HumanML3D and track the effective rank of hidden states along 10- to 60-second rollouts. The aim is to learn whether the collapse reported for video also appears in articulated motion, and whether it precedes rises in FID and foot skating. We also test whether it worsens with training length or with multi-task data, as the scaling paradox in the video paper would predict. We then measure the spectrum of $\Sigma_{l,k}$. If drift is nearly isotropic, the premise of structured augmentation fails, and we stop and report that.
The second stage builds the method on one unified masked-and-autoregressive transformer. That model handles text-to-motion, multi-sentence long motion, motion prediction, and in-betweening with keyframes placed 2 to 20 seconds apart. We add hand-goal reaching on CIRCLE, following COMET's task setup. We use a low-rank covariance of rank 16–64 per layer to keep sampling cheap, and we sweep the refresh interval $N$ and the scale ramp $\lambda$, since ISDA-style methods are known to be sensitive to scale.
The third stage probes the mechanism. We record attention mass on anchor tokens as a function of injected drift. We then retrain with anchors perturbed as well, to test whether clean anchors are what makes recovery possible.
Baselines
The naive reference is the same architecture trained with plain teacher forcing and no corruption. Every other result is read as an improvement over this.
The strongest existing competitor on HumanML3D is T2M-GPT with its τ=0.5 random code corruption, which reports FID 0.116 on HumanML3D (T2M-GPT). We will retrain it from the released recipe on our codebook, with the same step budget and a τ sweep. Beating it means equal or better single-clip FID and R-precision together with clearly better FID in long rollouts.
For long multi-sentence generation the reference is T2LM. It reports a sliding-scope FID of 0.440 on HumanML3D and a transition-scope FID of 1.389 on BABEL (T2LM). Beating it autoregressively would show that drift can be controlled without giving up global context. On CIRCLE, COMET with its reference-guided feedback is the natural competitor, but we found no reported result on Lacuna for COMET's success rate, distance-to-goal or foot skating on CIRCLE. We will therefore rerun it and report its inference latency alongside ours.
Three internal controls get identical data, steps and tuning trials. The first is VRR-style isotropic regularization applied at first, middle and last layers (VRR), whose only reported numbers are on VBench and not on motion. The second is isotropic Gaussian noise on context hidden states, matched in total variance to our perturbation. This is the key ablation: it isolates direction. The third is compute-matched rollout training, which feeds real self-generated context for part of each batch. It isolates whether an approximation of drift is enough.
Why not just…?
Why not just tune T2M-GPT's token corruption? Random code swaps teach recovery from errors the model rarely makes, such as jumping to an unrelated code. They do not cover errors it does make, which are small, correlated slips toward neighbouring codes, and the jitter T2M-GPT itself reports is consistent with that mismatch. The τ sweep in our baselines settles this directly. If the best τ matches our long-horizon numbers, then continuous structure buys nothing.
Why not just add isotropic noise to hidden states, or regularize their rank? This is the most serious threat, because both are simpler and VRR already works on video. If drift turns out to be nearly isotropic in stage one, or if the variance-matched isotropic control matches our method, then the structured covariance is unnecessary and we will say so. Our method wins only if concentrating perturbation on measured drift directions improves long-horizon FID or goal error at equal perturbation energy.
Why not just train on real rollouts with scheduled sampling? Real rollouts give the exact error distribution, so they are the gold standard, not a straw man. Their cost is that generating context sequentially breaks the parallel training that makes transformers cheap, and they only expose the model to errors of the magnitude it currently makes. Our method refreshes drift statistics only occasionally, and the $\lambda$ ramp extrapolates to larger errors. The compute-matched comparison decides the question. If rollout training wins at equal GPU hours, our contribution shrinks to a cheaper approximation and would need a speed argument to stand.
Why not just use a non-autoregressive decoder or a post-hoc corrector? T2LM shows that a local decoder avoids drift but cannot reach distant spatial targets. COMET and Thermalizer show that correctors work but add inference cost, and the smoothing they apply can dampen sharp movements. The in-betweening and CIRCLE experiments test whether a single autoregressive model can keep global reach without either compromise.
Proposed Experiments
Evaluation uses HumanML3D and KIT-ML for single-clip text-to-motion, measured by FID and R-precision. For long multi-sentence generation we use HumanML3D and BABEL, measured by sliding-scope and transition-scope FID. In-betweening and prediction use AMASS-derived clips with keyframes at controlled gaps, measured by keyframe joint error and foot skating. CIRCLE is used for goal reaching, measured by success rate and distance to goal. For every rollout we log effective rank per layer against time.
Two checks are decisive. First, at equal perturbation variance, drift-matched augmentation must beat isotropic hidden noise on 60-second sliding FID and on keyframe error at the widest gaps. Second, the advantage over the teacher-forced model must grow with distance to the nearest anchor, and it must largely vanish when anchors are also perturbed. If both hold, drift-shaped latent augmentation is a real mechanism, and long-range attention is the route through which the model corrects itself. If neither holds, simpler regularizers suffice for motion, and the diagnostic study still establishes whether representational collapse is what drives motion drift.
References
- Generating Human Motion from Textual Descriptions with Discrete Representations
- Mitigating Compounding Error via Video Representation Regularization
- Controllable Long-term Motion Generation with Extended Joint Targets
- Thermalizer: Stable autoregressive neural emulation of spatiotemporal chaos
- Implicit Semantic Data Augmentation for Deep Networks
- A Unified 3D Human Motion Synthesis Model via Conditional Variational Auto-Encoder∗
- T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences