Where Is Noisy Video Motion Worth Learning From? Tier-Specific Ambient Training for Text-to-Motion

Motion capture in a lab is clean but scarce. The main text-to-motion benchmark holds about 30 hours. Motion recovered from internet video is plentiful but corrupted: depth is ambiguous, limbs get occluded or flipped, and the body drifts and slides. This proposal asks a question nobody has measured. When a generator builds motion in stages, first the path through space, then the coarse body, then individual joints, is noisy video data worth the same at every stage? We expect it is not. A single camera sees limb angles reasonably well but recovers global depth and foot contact poorly. So video motion should be nearly as good as mocap for coarse body shape, and nearly worthless, or even harmful, for the root trajectory and fine contact detail. The automatically generated captions that come with video data have the same mismatch. They describe coarse actions plausibly and fine articulation badly.

We propose to train a hierarchical motion diffusion model that admits each data source into each stage only where that source is trustworthy. A diffusion model is a generator that learns to turn random noise into samples by repeated denoising. Here, "trustworthy" is measured, not assumed. For every stage and every diffusion noise level, a small classifier tests whether noisy video motion can be told apart from noised mocap. Video data trains that stage only at noise levels where the classifier cannot tell them apart. The video captions condition only the stages where text and motion actually correspond. The deliverables are two. The first is a measured per-stage "exchange rate" between clean and noisy motion. The second is a generator that uses the noisy data according to those rates.

Core Claim

Ambient diffusion, which trains diffusion models on corrupted data, has shown for images that noisy data teaches coarse, low-frequency structure while clean data is needed for fine detail. How Much is a Noisy Image Worth? reports that 10% clean plus 90% noisy ImageNet-64 reaches FID 1.68, against 1.41 for all-clean and 10.57 for the clean 10% alone. Ambient Diffusion Omni uses corrupted images only above a learned noise threshold $t_{\min}$. Above that threshold the forward noise drowns out the corruption. These two papers supply our method. Both assume pixel corruptions such as Gaussian noise, blur, or crops, and both use one threshold per dataset.

Our claim is that for human motion the threshold must be set per kinematic stage. The corruptions in monocular motion are structured and stage-dependent: depth drift lives in the root, and jitter and foot sliding live at the joints. These are exactly the structured, correlated corruptions that DrugOOD warns can be amplified rather than averaged away as data grows. Its result constrains the design. Admitting all noisy data everywhere should hurt, and some stage-and-noise cells must be closed to video data entirely.

Hierarchical generators motivate the stage split. GMD separates trajectory from pose. GUESS cascades four skeletal resolutions, from 1 node to 22 joints. Kimodo interleaves root and body denoising. All three train only on clean mocap. Kimodo explicitly gets scale from 700 hours of proprietary optical capture instead of from video.

Research Program

The model is a three-stage latent diffusion cascade in the style of GUESS. Stage one generates the root trajectory. Stage two generates a 5-node coarse body conditioned on stage one. Stage three generates all 22 joints conditioned on stage two. Coarse skeletons are computed from the full skeleton by averaging joints, so noisy video data can be downsampled without needing clean intermediate labels.

The data has two sources. The clean source is HumanML3D, built on AMASS mocap. The noisy source is the in-the-wild video portion of Motion-X, which provides 32.5K video-derived sequences with automatically generated captions. We use Motion-X with its smoothing intact and treat its leftover error as corruption.

For each stage $k$ and diffusion time $t$, we train a classifier to separate noised clean latents from noised video latents. We set the threshold $t_{\min}(k)$ as the smallest $t$ at which this classifier's accuracy falls to chance plus a tolerance. This is the same classifier-based rule Ambient-o uses, applied separately to each stage. The training loss for stage $k$ is then

$$\mathcal{L}_k = \mathbb{E}_{x\sim\text{clean}}\,\ell_k(x,t) + \lambda\,\mathbb{E}_{x\sim\text{video}}\,\mathbf{1}[t \ge t_{\min}(k)]\,\ell_k(x,t).$$

Here $\ell_k(x,t)$ is the ordinary denoising loss of stage $k$ on sample $x$ at diffusion time $t$. The indicator $\mathbf{1}[\cdot]$ admits a video sample only above that stage's threshold, and $\lambda$ weights the video term. A threshold set at the top of the noise range closes a stage to video data; a threshold at zero opens it completely.

The cross-modal gap enters in two ways. First, it limits where captions can be used. Text describes actions and coarse body configuration, and has no vocabulary for per-joint angular velocity. GUESS already reflects this: it weights text heavily at coarse stages and lightly at fine ones. Second, it gives us a test. We compute, for each stage, how well the video captions retrieve their own motions, using the retrieval set-up of Pyramidal Shapley-Taylor Learning. Video captions then condition only the stages where that retrieval is above a floor. At other stages, video samples are trained unconditionally, the way classifier-free guidance drops text.

The scientific output is a per-stage exchange rate $r_k$, defined as the number of video sequences that buy the same FID or foot-skating improvement at stage $k$ as one clean sequence. We estimate it by sweeping the clean fraction of HumanML3D (10%, 25%, 50%, 100%) against the amount of added video data.

To separate corruption type from domain shift, we also run a controlled version. We corrupt held-out AMASS motion with synthetic monocular-style errors: root depth drift, bone-length stretching, limb flips, and foot jitter. This lets us vary each error type independently and check that each stage's threshold responds to the error type that lives in that stage.

Baselines

The naive reference is the same three-stage cascade trained on clean HumanML3D only, at each clean fraction. Beating it is the minimum bar.

The strongest existing published method for this evaluation is GUESS, which reports FID 0.109 and R-Precision 0.503 on HumanML3D (GUESS). For context, T2M-GPT reports FID 0.116 (T2M-GPT) and GMD reports 0.212. GUESS is the right competitor because our backbone is its cascade: any gain must come from how data is admitted, not from architecture. We retrain GUESS ourselves with the same latent sizes, sampling steps, and compute as our model, and use its published number only as a sanity check on our reimplementation. At 100% clean data, success means matching 0.109 within seed variance while reducing foot skating. At 10–25% clean data, success means clearly beating clean-only training.

There is no reported result on Lacuna for any text-to-motion model trained on Motion-X video combined with HumanML3D and evaluated on HumanML3D. There is also no reported result found on Lacuna for clean-to-noisy exchange rates in motion. Kimodo's foot-skating figures, 3.87 cm/s versus 7.59 cm/s on Bones Rigplay, are on a different dataset and are cited only to motivate the stage split.

The pooled baseline trains the identical cascade on HumanML3D plus Motion-X video with no thresholds and full caption conditioning everywhere. This is what most practitioners would do.

Three ablations isolate the key ingredient, per-stage thresholds. The first uses a single global $t_{\min}$, which is Ambient-o transplanted directly to motion. The second uses per-stage hard masking, where each stage is either fully open or fully closed to video and no noise-dependent threshold is used. The third uses per-stage thresholds but full caption conditioning, which isolates the caption-routing component.

All variants share data, number of optimizer steps, and a fixed hyperparameter budget of eight runs each for tuning $\lambda$ and the learning rate.

Why not just…?

Why not just pool the video data and let scale average out the noise? The "noise cancels at scale" results, such as weakly supervised Instagram pretraining, concern uncorrelated label errors. Monocular depth drift is systematic: it appears in the same direction across sequences filmed with similar cameras. DrugOOD reports that systematic noise beyond a threshold outweighs the benefit of more data. The pooled baseline tests this directly. If pooling matches our method on FID, foot skating, and trajectory error, the per-stage thresholds are unnecessary, and we will report that.

Why not just add a data-source token so the model learns to tell video from mocap and generate in "mocap mode"? This is the most serious threat. It is cheap and might absorb most of the domain shift. We add it as a fourth baseline: the pooled cascade plus a source flag at every stage, with "mocap" set at sampling time. The difference in principle is that a flag lets corrupted root data still shape the representations shared across sources. Thresholds keep that data out of the noise levels where it is distinguishable. The comparison of trajectory error and foot skating between the flag baseline and our method settles this. If the flag wins or ties, we drop thresholds in favor of the flag and keep the exchange-rate measurement as the contribution.

Why not just discard video root trajectories and use video only for body pose? That is per-stage hard masking, and it may capture most of the gain if the root is simply unusable. Ambient-o's image results suggest that even badly corrupted data still carries coarse layout at high noise. For motion, that would mean a video trajectory still says "walking forward" even if its depth is wrong. The masking ablation measures whether those high-noise root updates add anything. If they do not, we will say so: hard masking is then the recommended recipe.

Why not just clean the data harder, with stricter confidence filters and stronger smoothing? Motion-X already smooths and fits its data, and residual error remains. Discarding data is also what ambient methods outperform: Ambient-o reports CIFAR-10 FID 2.44 with corrupted data versus 8.79 without it. We include a confidence-filtered pooled baseline at matched retained data, so whether filtering suffices for motion is tested rather than assumed.

Decisive Checks

All results use HumanML3D test FID, R-Precision, and diversity under the standard evaluator, plus foot skating and root-trajectory error. They are reported over three seeds at each clean fraction.

The central check is whether the measured thresholds and exchange rates actually differ across stages, with video worth most at the coarse body stage and least at the root. If the thresholds come out flat, the core hypothesis is false and the method reduces to Ambient-o.

The second check is whether, at 10–25% clean data, our model beats both the pooled baseline and the source-flag baseline on foot skating and trajectory error without losing R-Precision.

The third check is the synthetic-corruption study. Injecting depth drift should raise only the root stage's threshold, and injecting jitter should raise only the joint stage's threshold. If thresholds move in this way, the stage-specific mechanism is working as claimed. If they move together, it is not.