Amending the Plan, Not Resampling the Motion: Bottom-Up Corrections in Hierarchical Motion Generators

Most systems that generate 3D human motion from text or scene descriptions work in two stages. A planner first decides where the body's root travels, meaning the path of the pelvis through the room. A second model then fills in how the limbs move along that path. This split fixes many artifacts that single-stage models suffer from, but it adds a new failure: when the planner routes the body somewhere the limbs cannot physically go, such as through a chair leg or along a turn too sharp to walk, the second stage has no way to object. It produces a broken pose, and the error is silent. We propose letting the low-level model send a small, local, reasoned correction back up to the planner. Following the practice scholarly publishers use to fix wrong author affiliations, the correction does not quietly overwrite the original plan. It is issued as a separate, linked amendment that states what failed and why.

The payoff is twofold. At generation time, fixing a short window of the plan should cost far less than regenerating the whole sequence, and it should work better, because the planner is told the reason for the failure. Across many generations, the collected amendments form a structured, automatically produced record of where and why the hierarchy breaks, measured against scene clutter, turn rate, and contact density. That record is the kind of limitation disclosure motion papers rarely publish.

Core Claim

Cascading errors are the named weakness of staged motion generators. Contact-aware Human Motion Forecasting, which predicts joint–scene contact maps and then forecasts joints anchored on them, states explicitly that missed contacts in the first stage permanently corrupt the second. The same top-down structure appears in three other systems:

  • Guided Motion Diffusion uses a trajectory diffusion model followed by a motion diffusion model. A diffusion model is a generator that learns to turn random noise into samples by gradual denoising.
  • SceMoS uses a bird's-eye-view planner followed by a heightmap-conditioned motion tokenizer. - Scene-Aware Generative Network uses separate trajectory and pose generators.

In all four, information flows only downward. SyncDiff does add an explicit synchronization step at inference time. That step aligns multiple bodies with one another, not levels of the hierarchy, and the authors themselves call it time-consuming.

Our claim is that a bottom-up amendment channel is a cheap way to repair cascading errors when failures are local in time. The channel has three parts:

  • a failure detector at the pose level;
  • a reason code drawn from a fixed taxonomy;
  • a planner that re-plans only the affected window, conditioned on that code.

The amendment log produced along the way is a measurable failure map that can be checked against held-out scenes.

The affiliation-correction literature plays a design-constraint role here, not a methodological one. Correction notices such as Correction: Instant3D, which reassigns two authors from Singapore Management University to Sea AI Lab, and Correction to: IGFM keep the original record intact. They add a separate amendment that names the defect and link it in both directions to the original. We borrow exactly that discipline: no silent overwrites, an explicit defect statement, and two-way links. We borrow it because it is what turns local repairs into analyzable data. The analogy is structural only, and we do not claim more for it.

Research Program

We build on a SceMoS-style pipeline because its planner and local generator are already separate, token-based, and fast. The file reports that SceMoS generates 80 frames in about 8 seconds. The planner emits root-trajectory tokens autoregressively over a bird's-eye-view image of the scene. The local tokenizer produces pose tokens conditioned on nearby 2D heightmaps.

The failure detector. We add a lightweight critic that scores each window $w$ of $L$ frames with a feasibility cost $$c(w) = \alpha\, P(w) + \beta\, S(w) + \gamma\, D(w),$$ where the three terms are:

  • $P(w)$, the maximum body–scene penetration depth;
  • $S(w)$, foot-skating distance during detected ground contact;
  • $D(w)$, the gap between the root displacement implied by the generated limb motion and the planned root path.

The weights $\alpha, \beta, \gamma$ are fixed on validation data. $D(w)$ is the term specific to hierarchical models. It detects when the pose model has in effect refused the plan.

The amendment. When $c(w)$ exceeds a threshold, the critic emits an amendment. The amendment records the window span, a reason code (collision with a named scene region, infeasible turn, missed contact affordance, or skating), the original plan tokens, and the scene location of the failure.

The re-planning step. The planner regenerates only tokens in $[t-k, t+k]$ around the failing window. The prefix and suffix are held fixed as anchors. The planner is conditioned on the reason code and on a mask that marks the failure location on the bird's-eye-view image. The pose model then regenerates only that window. The amendment also stores feasibility before and after the fix, and every final frame points to the plan version that produced it.

Training. The planner learns to use reason codes during fine-tuning with synthetic amendments. We perturb ground-truth trajectories to create failures, label them with the critic, and train the planner to recover the true segment given the code.

The failure map. Across a stress suite, we regress amendment frequency and residual failure on scene descriptors: free-space fraction along the path, obstacle count within a body radius, curvature of the requested path, and number of required contacts. The fitted map is itself a deliverable, a limitation disclosure produced by measurement rather than by the authors' recollection. It responds directly to the missing failure taxonomy described in the file's unspecified-limitations material.

Baselines

Naive reference. The unmodified two-stage pipeline with no feedback. On TRUMANS, SceMoS reports FID 0.31 and Contact Score 0.98. FID, Fréchet Inception Distance, measures how close generated motions are to real ones in distribution. Our reproduction must land near these values before any comparison counts. Because those numbers show near-saturation on standard TRUMANS, the decisive comparisons happen on the stress suite described below.

Strongest competitor: best-of-N resampling. Generate $N$ full sequences and keep the one with the lowest total critic cost. This baseline needs no architectural change and uses the same critic, so it isolates the value of local, reasoned repair. We match wall-clock compute on the same GPU: $N$ is chosen so that its total time equals the mean time of our method, including its amendment rounds. The critic and thresholds are identical for both.

Top-down guidance. Gradient-based guidance on the trajectory in the style of Guided Motion Diffusion, adapted as critic-gradient steering during planning. GMD reports FID 0.212 against MDM's 0.556 on HumanML3D, which shows top-down guidance is already strong. Its guidance can only use a cost defined at the plan level. It cannot see pose-level refusal, which the $D(w)$ term captures.

Cross-benchmark reference. For forecasting-style evaluation on GTA-IM and PROX, Contact-aware Human Motion Forecasting reports global-path MPJPE (mean per-joint position error) of 108.4mm against 147.4mm on GTA-IM, and 192.2mm against 236.3mm on PROX. We use these as references for whether amendments preserve path accuracy.

Ablations. We run two:

  • Silent local resampling: the same window-level regeneration, but with no reason code and no failure mask. This isolates the key ingredient, telling the planner why the window failed.
  • Pose-only repair: re-running only the pose model in the failing window without changing the plan. This tests whether the plan really needs changing.

What counts as beating the baselines. Our method beats best-of-N if, at matched wall-clock time, it achieves lower penetration and skating and a higher task-success rate on the stress suite, with FID no worse than the baseline's.

Why not just…?

Why not just sample several full motions and keep the best one? This is the real threat. If failures are spread across the whole sequence rather than local in time, best-of-N at matched compute may match or beat local amendment. The compute-matched comparison settles this directly. We will also report the empirical distribution of failure-window lengths. If most failures span more than about half the sequence, the premise is wrong, and we would report local amendment as unhelpful in that regime.

Why not just use gradient guidance on the planner, as GMD does? Guidance steers the plan toward low cost under a cost the planner can evaluate. The characteristic cascading failure, however, is visible only after the pose model tries to execute the plan. That is the disagreement term $D(w)$, and it is not differentiable through discrete pose tokens. The guidance baseline measures how much of the gap plan-level costs close on their own.

Why not just clean up the final motion with inverse kinematics or a physics filter? Post-hoc cleanup can remove skating. It cannot reroute a path that passes through furniture without large pose distortion. The pose-only repair ablation is the in-program version of this alternative. We expect it to fix skating-coded failures and fail on collision-coded ones. If it fixes both, the planner-level amendment is unnecessary.

Why keep an amendment log instead of just reporting a failure rate? A single failure rate cannot say where the model breaks. The log attributes each failure to a scene configuration and a reason. Its value is tested by whether a failure map fit on training scenes predicts failures on held-out scenes better than a scene-agnostic base rate.

Proposed Experiments

Stress suite. Standard TRUMANS appears saturated. We therefore build a stress split by inserting additional obstacles from the benchmark's own object set into held-out TRUMANS scenes and by requesting paths with rising curvature. The ground truth is the task specification, not a motion capture recording, so evaluation uses:

  • penetration depth;
  • skating distance;
  • goal-reaching and contact success;
  • the SceMoS Contact Score;
  • FID against real TRUMANS motion;
  • wall-clock latency.

We also evaluate on HUMANISE, and on GTA-IM and PROX with MPJPE, to check that amendments do not degrade path accuracy relative to the 108.4mm and 192.2mm references above.

First decisive check. At matched compute, does reason-coded local amendment beat both best-of-N and silent local resampling on stress-suite penetration and success? Beating best-of-N shows that local repair pays. Beating silent resampling shows that the reason code carries information.

Second decisive check. We fit the failure map on half of the stress scenes. On the other half, it must predict per-scene amendment rates with rank correlation clearly above a base-rate predictor. If it cannot, the log is bookkeeping, not a limitation disclosure.

Third check. We report amendment rate against obstacle density, both as a curve and as a stated threshold. This is the quantified operational boundary that SceMoS acknowledges only qualitatively when it says the method is restricted to static scenes without fine manipulation. No reported result was found on Lacuna for failure rates of any hierarchical motion generator as a function of scene clutter, so this curve would be new.