Soft Syntax Coordinates for Compositional Text-to-Motion

Text-to-motion generators turn a sentence into a 3D skeleton animation. They break in a predictable way on compositional commands such as "walk forward, then stop and wave with the left hand." They drop an action, reorder actions, or move the wrong limb. The most successful fixes run the prompt through an automatic parser first, either a semantic role parser or an LLM that splits the prompt into events, and then condition the generator on that structure. These fixes buy compositionality at a price. Whatever the parser gets wrong is handed to the generator as fact. If "with the left hand" attaches to "stop" instead of "wave," the model faithfully animates the mistake.

We propose to keep the parser's structure but carry its uncertainty along with it. Each prompt gets several candidate parses with probabilities. From each parse we compute tree-based position signals, such as how far apart two words are in the tree and which clause a word belongs to. We average these signals by parse probability and inject them only as soft biases on attention, never as hard routing edges. The biases act inside a generator that models every joint at every frame. When the parser is confident, the model sees a sharp tree. When it is torn between readings, the bias blurs toward the plain word-order behavior of a flat model. The core claim is that uncertainty-weighted tree biases retain most of the compositional gain of parser-based conditioning. They should also lose far less when the parse is wrong, which is exactly where current structured methods fail.

Why this, and not the obvious combination

The obvious idea at this intersection is a "dual-tree" model: tree positional encodings for both the sentence's syntax and the skeleton's bone hierarchy, joined by cross-attention. We keep it only as an ablation, for a reason a reviewer would raise immediately. The skeleton is fixed (22 joints in HumanML3D). For a fixed tree, a learned per-joint embedding can represent anything a tree coordinate can. MoGenTS already uses per-joint positions on its time-by-joint token grid and reports HumanML3D FID 0.033. A kinematic-tree encoding might still help as a relative attention bias, but that is a modest architectural tweak, not a research question.

The sentence side is different. Its tree changes with every prompt and is predicted, not given, so here the open problem is real. Three findings shape the design.

GraphMotion builds a three-level semantic graph (whole motion, actions, action attributes) from a semantic role parser and runs graph attention over it inside a coarse-to-fine diffusion model. It reports HumanML3D FID 0.116 and Top-1 R-Precision 0.504. Its authors name reliance on the parser as a core limitation. This paper motivates the proposal: structure helps, and hard structure is exposed to parser error.

Event-T2M uses an LLM to split prompts into events and cross-attends to them as a flat sequence. It introduces HumanML3D-E, which stratifies test prompts by event count, and reports FID 0.265 on the four-or-more-event subset against 0.418 for MoMask. It has the same upstream exposure and discards any nesting, such as which modifier belongs to which verb.

Do We Really Need GNNs with Explicit Structural Modeling? constrains the design. It shows that message passing along parsed dependency edges is fragile under structural noise: a message-only module degrades Llama-3 probing by 5.47 F1, and plain MLPs beat full GNNs. We read this as a verdict against hard routing, not against structure. This is why our tree signal enters only as an additive attention bias that the model can learn to down-weight, rather than as the edges information must flow along.

Two further papers supply the method. Rethinking Positional Encoding in Tree Transformer for Code Representation encodes each tree node by its recursive (sibling index, sibling count) path and adds global and local structural biases to self-attention, improving JavaScript150k completion from 63.34% to 66.48%. It assumes compiler-exact trees, which is the assumption we relax. Dependency Transformer Grammars avoids an upstream parser by generating syntax jointly with text, using stack-depth positions. That route is unavailable to a motion generator, which never generates the text. Encoding Sentences with Graph Convolutional Networks for Semantic Role Labeling down-weights noisy arcs with learned gates. Its gates are driven by the network's own guess about each arc; ours are driven by the parser's calibrated disagreement among whole trees.

Core Claim

Consider a joint-level generator whose text conditioning receives tree structure only as soft attention biases, averaged over the parser's k-best parses. Compared with the same generator given the parser's single best tree, it should (a) match it on prompts the parser gets right, (b) degrade substantially less as parse errors are injected, and (c) beat both flat conditioning and single-tree conditioning on multi-event and limb-attribution prompts.

Research Program

Parser input. A dependency parser that outputs scores returns its top $K$ parses $\{\tau_k\}$ for each prompt, with renormalized probabilities $\pi_k$. We use $K = 8$ and state the parser we use; the choice is not fixed by prior work.

Signal 1: soft tree distance in the text encoder. For text tokens $u, v$, let $d_k(u,v)$ be their path length in parse $\tau_k$. The text encoder's self-attention gets the bias

$$b_{uv} = \sum_{k=1}^{K} \pi_k \, g_h\big(d_k(u,v)\big),$$

where $g_h$ is a learned scalar per attention head over bucketed distances. We also add the recursive (sibling index, sibling count) coordinates of Rethinking Positional Encoding in Tree Transformer, averaged the same way. When all parses agree, $b_{uv}$ is an exact tree bias. When they disagree, it is a blend that the heads can learn to ignore.

Signal 2: soft clause membership in cross-attention. For each verb $c$ and token $u$, let $m_{uc} = \sum_k \pi_k \mathbf{1}[u \in \text{subtree}_k(c)]$ be the probability that $u$ belongs to $c$'s clause. The cross-attention from motion token $(t, j)$ (frame $t$, joint $j$) to text is factored in two steps. First, a small attention over clauses picks which verb frame $t$ is executing, with a learned bias toward monotone clause order over time. Second, attention within the chosen clause reaches tokens weighted by $m_{uc}$. This is where a misattached "left hand" would do damage. With soft membership, a doubtful modifier is shared between candidate verbs instead of being assigned wholesale to the wrong one.

Backbone. We reuse the joint-level time-by-joint token map and alternating spatial/temporal attention of MoGenTS, with its stochastic masked-token sampler. It has the strongest reported fidelity, and joint-level tokens are necessary for limb attribution to be expressible at all.

Training. We train on HumanML3D using the same parser's k-best output on training captions. Parse noise at training time therefore matches test conditions.

Diagnostic extension. The original four directions include the claim of On Error Propagation of Diffusion Models that errors early in a denoising chain do not shrink. By analogy, a conditioning error should persist across iterative sampling steps. We test whether that analogy holds for masked iterative decoding by measuring, per step, how far generations from a corrupted parse and from a correct parse have diverged. If divergence locks in within the first few steps, a second step is justified: re-scoring the $K$ parses mid-sampling by how well each explains the partially generated motion. That step is conditional on the measurement, not assumed.

Baselines

Flat conditioning (the "do nothing" reference). This is the MoGenTS backbone with its standard text encoding and no parser. It shows what structure adds at all. We report the published HumanML3D FID of 0.033 from MoGenTS and retrain it on our pipeline with identical data, steps, and tuning budget. MoGenTS reports no HumanML3D-E numbers, so we evaluate it on that split ourselves.

Event-T2M, the strongest existing method on complex prompts. Event-T2M reports FID 0.265 on the four-or-more-event subset of HumanML3D-E. We will rerun it on our splits under the same parse-corruption protocol, corrupting its LLM event lists with analogous error types. On clean prompts, beating it means lower FID and higher Top-1 R-Precision on that subset. The more important criterion is a flatter degradation curve under corruption.

GraphMotion. GraphMotion reports FID 0.116 and R-Precision 0.504 on HumanML3D. It is the direct hard-graph competitor and can be run from the released GraphMotion code with its parser swapped for ours. This lets us isolate the effect of hard routing from the effect of parser quality.

Key ablations. The ablation that isolates the proposal's main ingredient is our own model with $K = 1$: the same tree biases and clause-factored cross-attention, but only the single best parse. Every gain attributed to uncertainty must show up as a gap between $K = 8$ and $K = 1$. A second ablation replaces the bias with graph-attention message passing over the same parses, testing the fragility finding of Do We Really Need GNNs? in the generative setting. A third adds an MLP with matched parameter count and no structure, as that paper recommends. A fourth swaps learned joint embeddings for kinematic-tree coordinates.

All baselines and ablations get the same number of training steps and the same hyperparameter-search budget, and are compared at matched sampling steps.

Why not just…?

Why not just use a better LLM decomposer, as Event-T2M does? This is plausible for clean event splitting, and our experiments will show it if so. An LLM plan is still a single hard output. Event-T2M's own numbers leave room: FID on four-or-more-event prompts is 0.265, versus 0.033 overall for the best flat model, although these come from different models and evaluation sets. Its decomposition also adds about 1.6 s per prompt. Most importantly, a flat event list has no place to record which modifier belongs to which event. The limb-attribution test below directly measures whether that matters.

Why not just train the single-best-parse model with random parse corruption? This is the real threat. Noise augmentation can teach a model to distrust structure globally. It cannot tell the model which prompt's parse is shaky, whereas the parser's k-best disagreement can. We include an augmented $K = 1$ variant trained at several corruption rates. If it matches $K = 8$ on the corruption curves, then the uncertainty signal adds nothing and we would report that. If our advantage concentrates on prompts where the parser's top parses disagree, the per-prompt signal is doing the work.

Why not rely on a large text encoder's implicit syntax? The MLP result suggests pretrained models already hold much syntactic information. Action collapse on multi-event prompts, documented by Event-T2M, suggests that information does not reach the motion generator. The flat baseline, run with a large encoder, settles this directly.

Why not a tree encoding for the skeleton? For a fixed 22-joint skeleton it is nearly equivalent to learned joint embeddings. We expect at most a small effect and run it only as an ablation.

Proposed Experiments

Go/no-go check. Before building anything, measure how often the parser's top two HumanML3D-E parses differ on verb or modifier attachment. The file reports no parser accuracy on motion captions. If disagreement is rare on four-or-more-event prompts, the proposal's headroom is small and we will say so.

Standard benchmarks. We evaluate on HumanML3D, KIT-ML, and all HumanML3D-E strata, using FID, Top-1/2/3 R-Precision, Diversity, and Multimodality.

Decisive test 1: parse corruption. At test time we inject controlled errors into the parse (or into Event-T2M's event list): modifier reattachment, conjunct swaps, and clause merges, at rates from 0 to 50%. We plot R-Precision and FID against the corruption rate. The claim fails if our $K = 8$ curve is no flatter than $K = 1$ and GraphMotion.

Decisive test 2: limb attribution. We build a templated set of attachment-ambiguous prompts such as "wave with the left hand while stepping to the right." The automatic score is the fraction of samples in which the named limb carries the most joint-velocity energy during the named action's time segment. We also hand-parse about 500 multi-event test captions, so results can be split by whether the parser was right.

Diagnostic test. We measure the per-step divergence curve between generations from corrupted and clean parses. This decides whether mid-sampling parse re-scoring is worth building.