Sampling Physically Constrained Motion Tokens Without Getting Stuck

Text-to-motion generators such as MoMask turn human movement into sequences of discrete codes ("tokens") and learn to produce those codes from motion-capture data. They produce realistic motion, but they cannot be told to obey a new physical requirement at sampling time. Examples include "keep planted feet still," "do not go through the floor," and "end the walk inside this one-metre circle." Adding such a requirement turns generation into a sampling problem: draw motions from the pretrained model's distribution reweighted by $e^{-\lambda U}$, where $U$ is a physics penalty computed on the decoded motion and $\lambda$ sets how strictly it is enforced. Statistical physics has studied this kind of problem for decades. It is known to break simple samplers when constraints are strong. Local Markov chain Monte Carlo (MCMC), which changes one variable at a time, suffers critical slowing down: successive samples stay correlated for a very long time, because any single-variable change conflicts with long-range order. Motion tokens have exactly this long-range coupling. A goal constraint at the last frame depends on every earlier step of the walk, and a foot-contact constraint spans the boundary between neighbouring tokens.

We propose to sample this reweighted distribution with a neural sampler, a network trained only from the penalty function to produce nearly independent samples in one pass. Our sampler is built on the masked-diffusion neural sampler of MDNS, but it uses the pretrained motion model itself as the reference process. Every sample carries an exact importance weight, so statistics of the constrained distribution stay asymptotically unbiased. The work has two parts. We first measure whether critical slowing down actually occurs in motion-token space as $\lambda$ and constraint tightness grow. We then test whether the learned sampler keeps its effective sample size where cheap alternatives, such as generating many motions and keeping the good ones, collapse.

Core Claim

Constrained motion generation is a tilted sampling problem. The masked-unmasking procedure MoMask already uses can be recast as a stochastic optimal control problem with an exactly computable path weight. Let $\omega$ denote one full unmasking trajectory: the order in which positions are revealed and the token chosen at each reveal. Let $P_0(\omega)$ be the probability that the frozen pretrained sampler produces $\omega$, let $x_\omega$ be the final token sequence, and let $D$ be the frozen quantized-autoencoder decoder. The target is

$$P^*_\lambda(\omega) = \frac{1}{Z_\lambda} P_0(\omega)\, e^{-\lambda U(D(x_\omega))},$$

where $Z_\lambda$ is the unknown normalizer. The marginal of this target over final motions is exactly the pretrained model reweighted by $e^{-\lambda U}$. A trainable controlled sampler $P_\theta$ yields the importance weight

$$w(\omega) = \frac{P_0(\omega)\, e^{-\lambda U(D(x_\omega))}}{P_\theta(\omega)}.$$

Both path probabilities are products of categorical choices along the trajectory, so the weight is tractable even though neither model gives a marginal likelihood over motions. This construction is what makes the approach exact rather than heuristic.

The empirical claim has two conditions. First, token-level MCMC shows a sharply growing autocorrelation time as the constraint tightens. Second, best-of-$N$ resampling loses effective samples roughly as $1/(1+\chi^2(\pi_\lambda \| p_0))$, the standard importance-sampling identity, which degrades quickly once the constraint excludes most of the prior's mass. Under those conditions, a sampler trained with MDNS's log-variance or weighted denoising cross-entropy losses retains a usable effective sample size, balances distinct solution modes, and keeps motion quality.

The proposal differs from the closest work in specific ways. MDNS samples Ising and Potts spin grids from a pure energy with an uninformative all-mask reference. It has never been applied to learned codebooks, temporal sequences, text conditioning, or a data-driven prior. MoMask unmasks tokens with a confidence heuristic trained by supervised cross-entropy and has no mechanism for sampling an unnormalized target. The file's own audit reports no paper connecting the two lines of work.

Research Program

The pieces we build on. The prior comes from MoMask. It uses a residual vector-quantized autoencoder: a base codebook layer plus refinement layers that encode what the base layer missed. A bidirectional masked transformer fills in base tokens over about 10 steps, and a residual transformer adds refinements over about 5 more. We keep the decoder and residual transformer frozen and fill the residual tokens deterministically. The decoded motion is then a function of the base tokens alone, and control acts only on the base-token unmasking.

MoMask's confidence-ordered unmasking with remasking makes $P_0(\omega)$ awkward to compute. We therefore first switch the prior to random-order stochastic unmasking with the same network, and we report the FID cost of that change. This is the first go/no-go point. If the reference prior itself drops far below MoMask quality, the program needs a different reference.

Energies. The penalties are differentiable-free functions of the decoded joints. They include a foot-skate term (velocity of feet whose height is below a contact threshold), a ground-penetration term, and goal and waypoint terms (squared distance of the root to a target region at a given frame, with a shrinking radius). The goal constraints matter most scientifically. A far or awkwardly placed goal can often be reached in qualitatively different ways, such as turning left or right around a target, or stepping versus striding. This recreates the multimodal, long-range coupling under which local samplers freeze.

The controlled sampler. It is a copy of the masked transformer, initialized at the prior. Its output rates are corrected multiplicatively, which corresponds to the exponential-tilt control that DCMC links to Schrödinger-bridge optimality. Training uses no new motion data, only $U$. We compare MDNS's three objective families: relative entropy, log-variance, and weighted denoising cross-entropy with importance-weighted replay. Code comes from yuchen-zhu-zyc/MDNS. At test time we use self-normalized importance resampling with $w(\omega)$. By the argument of Nicoli et al. and Kanwar et al., this keeps expectations under the constrained distribution asymptotically unbiased even when the sampler is imperfect.

The diagnostic study. It stands on its own, and it transfers the physicists' measurement discipline to motion. For a fixed set of HumanML3D prompts, we sweep $\lambda$ and the goal radius. At each point we measure the integrated autocorrelation time of token-level Metropolized Gibbs sampling, which uses the prior's conditionals as proposals and $e^{-\lambda U}$ in the accept step. We also measure the effective sample size per unit compute for best-of-$N$ and for the learned sampler.

Baselines

The do-nothing reference is unconstrained MoMask sampling on HumanML3D. MoMask reports an FID of 0.045 there. It sets the quality ceiling any constrained sampler should approach, and it shows how often the constraints are violated when nothing is done. We also report T2M-GPT as a reference point. That paper reports FID 0.116 and Top-3 R-precision 0.775 for itself, while MoMask reports T2M-GPT's FID as 0.141. We will quote both and rerun it ourselves.

The strongest existing competitor is sampling-importance-resampling on MoMask. We draw $N$ samples from the prior, weight each by $e^{-\lambda U}$, and resample. It is exact as $N\to\infty$, it needs no training, and it is what a practitioner would try first. For a fair comparison, both methods get the same total budget of transformer forward passes, with the neural sampler's training cost amortized over the evaluation prompts and reported separately. Both use the same frozen decoder and energies and are evaluated on the same prompt and constraint grid.

The second strong competitor is token-level MCMC with the prior's conditionals as the proposal. It is exact and cheap per step, and it is the method critical slowing down should hurt. We run it for the same wall-clock time and report its integrated autocorrelation time.

The third competitor is post-hoc continuous optimization. We take the MoMask output and run gradient descent on $U$ in joint space, with a proximity penalty that keeps the motion close to the original.

The key-ingredient ablation trains the same controlled sampler on the same target with the relative-entropy objective only. That objective is the reverse-KL loss that KL-regularized reinforcement-learning fine-tuning also optimizes, and it is known to seek a single mode. If the log-variance or weighted-cross-entropy versions do not beat it on mode balance and effective sample size, the physics-sampler machinery adds nothing over reward fine-tuning.

Beating the baselines has a precise meaning. At constraint strengths where resampling's effective sample size per budget falls below a few percent, the learned sampler must reach several times higher effective sample size. It must also match the exact reference's mode proportions within statistical error and keep FID within a small margin of the prior's. There is no reported result on Lacuna for constraint-tilted sampling of motion tokens, so every number here will be new.

Why not just…?

Why not just generate many MoMask samples and keep the ones that satisfy the constraint? This is the most serious threat, and for mild constraints it will probably win, since it costs no training. The question is purely where the crossover lies. The importance-sampling identity predicts that its effective sample size decays with the chi-squared divergence between the constrained and unconstrained distributions, and that divergence grows fast as the goal radius shrinks. The $\lambda$/radius sweep measures this directly. If resampling stays competitive across all practically relevant constraint strengths, we will report that and drop the learned sampler. The diagnostic study would remain as the contribution.

Why not fix the decoded motion with continuous optimization or a physics clean-up pass? Local clean-up can remove foot sliding. It cannot choose between discrete solution modes, such as turning the other way to reach a goal. It also moves the motion off the codebook's learned manifold, which the tokenization literature (BEST, MeshTalk) credits with filtering out jitter and avoiding over-smoothed average poses. Most importantly, it samples no defined distribution, so its outputs carry no weights and no unbiasedness. We measure its constraint satisfaction, FID, and mode proportions alongside our sampler to test this empirically.

Why not use continuous diffusion with guidance instead of discrete tokens? Guidance is a biased heuristic with no exactness guarantee. The continuous backbones are also weaker on this benchmark: T2M-GPT reports FID 0.630 for MotionDiffuse and 0.544 for MDM, against 0.116 for T2M-GPT and 0.045 for MoMask. We therefore treat the discrete prior as the better starting point and do not rebuild on a continuous one.

Why not just fine-tune MoMask with reinforcement learning on $-U$? Its target is the same tilted distribution, which is why it appears as the relative-entropy ablation rather than as a separate method. The difference we test is whether off-policy log-variance and weighted-cross-entropy training avoid the collapse onto a single mode that reverse-KL training shows. MDNS reports this avoidance on Potts models near criticality, but on motion it is untested.

Proposed Experiments and Decisive Checks

All experiments use HumanML3D and report FID, Top-3 R-precision, diversity, constraint energy and satisfaction rate, effective sample size per forward pass, and integrated autocorrelation time for MCMC. KIT-ML serves as a secondary transfer check; the file lists no reported number on Lacuna for KIT-ML.

For bias, the ground truth at moderate constraint strengths is a very large resampling run on the prior. Against it we compare expectations of the energy, root-endpoint histograms, and mode proportions for multimodal goals, following the Ising/Potts protocol of comparing correlation functions to exact references.

First decisive check. Does critical slowing down exist here? If token-level MCMC's autocorrelation time stays small as constraints tighten, the premise fails and we say so.

Second check. Does random-order unmasking keep the prior close to MoMask quality?

Third check. Where does the effective-sample-size curve of resampling cross the learned sampler's, and does that crossover fall at constraint strengths people actually need, such as goal radii of tens of centimetres?

Fourth check. Does the ablation show log-variance or weighted-cross-entropy training preserving mode balance where relative entropy collapses?

We also watch for the degradation near critical regimes that both DCMC and MDNS report at critical temperatures. If it shows up at the tightest constraints, we will test annealed training over $\lambda$, in the spirit of the multilevel and annealed samplers of Multilevel Generative Samplers and flow-field Monte Carlo, before concluding that the approach fails there.