Locate, Mask, Regenerate: Test-Time Repair for Multi-Event Motion Generation

Text-to-motion generators produce 3D human skeletal animation from a sentence. They are good at "a person walks forward" and poor at "a person walks forward, stops, picks something up with the left hand, then turns and sits." On long instructions they drop actions, swap their order, or blur two of them together. When that happens today, the usual fix is to sample the whole sequence again and keep the best attempt. That spends compute on frames that were already fine and throws away good work along with the bad. This proposal spends extra inference compute on targeted repair instead. A checker finds which instructed event failed and where in the body and timeline it failed. The generator then regenerates only that region with a mask, keeping the rest of the motion fixed as context.

The mechanism is inpainting with a diffusion model, a generator that learns to turn random noise into samples by gradual denoising. Inpainting means freezing some joints and frames and letting the model fill in the rest so it stays consistent with them. Motion diffusion models already support this without retraining. The open question is whether a cheap, localized "find the failure, re-mask it, regenerate" loop gets more out of a fixed compute budget than whole-sequence resampling or restarting from the last good prefix. A second question is which parts of that loop actually need a language-model agent.

Core Claim

For multi-event motion prompts, the useful unit of test-time search is the failed event's spatiotemporal window, not the whole rollout and not the prefix. A loop that (a) splits the prompt into checkable events with expected order and involved body parts, (b) verifies each event window with short, perception-only queries, and (c) repairs only failing windows by masked re-noising should beat matched-compute best-of-N and prefix-restart search on event completion. It should do this without hurting overall realism. We also make a narrower, falsifiable claim about where the gain comes from: localization, not multi-agent deliberation. We test this directly by replacing every language-model role with a non-agentic counterpart. If the non-agentic version matches, the paper's contribution narrows to localized repair, and we will report it that way.

Relation to Prior Work

The closest work is Temporal Backtracking Search. It runs a generate-verify-restart loop for video diffusion, finds the earliest failing frame, and restarts generation from the clean prefix before it. On Keys-and-Doors Strict-OOD it reached 22.7% exact match versus 0.7% for matched-budget best-of-N. That paper motivates our approach, because it shows that process-level localization beats indivisible rollouts. We differ in three ways:

  • Its restart discards everything after the failure. Ours regenerates a bounded window conditioned on context on both sides, which matters when the later events were already correct.
  • Its masks are purely temporal. Ours can also be spatial, for example only the left arm during frames 60–90.
  • It relies on simulator state or a trained predictor as the verifier. Motion has no simulator for "did they pick something up," so the verifier itself becomes a research object.

Inference-time scaling of diffusion models through classical search combines depth-first backtracking that re-injects noise with Langevin refinement under fixed reward functions. UnMaskFork argues the opposite for masked text diffusion: deterministic branching beats stochastic resampling there (28.0% vs 19.0% Pass@1 on LiveCodeBench). We borrow both search styles as candidate repair operators and test which one wins inside a masked motion window, a question neither paper addresses.

Three papers supply the masking machinery. Human Motion Diffusion Model (MDM) predicts clean motion directly and supports training-free joint and frame inpainting. FLAME introduces diffuse-then-denoise editing under a binary mask. EDGE edits individual limbs while preserving the rest of a dance. All three require a human to choose the mask. We choose it automatically from a verifier's output.

Two papers supply the event decomposition. Event-T2M uses a language model to split prompts into events and conditions the diffusion model on them. Motion-R1 writes decomposed reasoning before generation and trains with reinforcement learning. Both are open-loop: once the denoiser drops an event, nothing goes back to fix it. We reuse their decompositions as the list of claims to verify.

Two papers constrain the verifier design. More Thinking, Less Seeing? finds that longer chains of thought in multimodal models shift attention from visual tokens to generated text and increase hallucination. We therefore forbid long-form reasoning in the checker. Each query is a single, short, perception-level question about one window. Mitigating LVLM Hallucination via Multi-agent System reports that badly phrased supervisor prompts propagate errors downstream. For that reason, structured event lists rather than free-form debate drive our pipeline.

Two papers motivate role separation without dictating it. Beyond Lines and Circles separates language solvers from tool solvers and validators, and MACM separates a hypothesizing Thinker from a verifying Judge. Both show gains from keeping verification apart from generation, but only with deterministic tools. Our "tool" is a stochastic generator.

Research Program

The system has three components, deliberately kept thin.

Planner. The planner turns the prompt into an ordered event list. Each event carries a rough time span and the body parts it involves. This is Event-T2M-style segmentation with body-part tags added.

Checker. After an initial generation, the checker scores each event window. It has two channels:

  • A programmatic kinematic channel measures foot skating, ground penetration, and jitter using forward kinematics. No language model is involved.
  • A semantic channel asks whether the window shows the event. We build it two ways. One uses a motion-text retrieval model's similarity between the window and the event text. The other renders the window and asks a vision-language model a one-sentence yes/no question.

Repairer. The repairer turns the checker's output into a mask covering the failing window, a short boundary margin, and only the tagged joints. It then re-noises the masked region to an intermediate noise level and denoises with the unmasked motion clamped. The loop runs until all events pass or the budget is spent.

The analysis has three threads:

  1. Repair operator. We vary noise re-injection level, stochastic resampling, and deterministic re-denoising with a perturbed event condition. This settles, for motion, the conflict between noise re-injection Inference-time scaling of diffusion models through classical search and deterministic branching UnMaskFork.
  2. Mask shape. We compare temporal-only, spatial-and-temporal, and prefix-to-end masks. The prefix-to-end mask is the restart strategy of Temporal Backtracking Search.
  3. Verifier quality. We plot repair gains against measured checker precision, because the cited search papers warn that search inherits its verifier's errors.

All compute is counted in denoiser function evaluations. Language-model tokens are reported separately so that agentic overhead is visible rather than hidden.

Baselines

Do nothing. The reference point is single-sample generation from the base model. The primary base is MDM, chosen because its clean-motion prediction gives training-free inpainting. Event-T2M is a second base if its released model can accept clamped context. The proposal is evaluated on top of each base, so gains are measured against that same base.

Best-of-N. This draws N full sequences and selects by the same semantic checker averaged over events. N is set so total function evaluations equal our loop's. This is the default way to spend test-time compute, and in Temporal Backtracking Search it is the comparison that collapsed to 0.7% out of distribution. Beating it means higher event-completion rate at equal function evaluations, with realism (FID) no worse.

Prefix restart. This is our reimplementation of Temporal Backtracking Search's strategy on motion. It uses the same checker to find the earliest failing frame and regenerates everything after it, under the same budget. It is the strongest existing method for the specific question of localized test-time search. Beating it requires a gain on prompts where the failure is mid-sequence and later events were correct, plus no loss elsewhere.

Train-time decomposition, no search. The strongest existing number on multi-event prompts is Event-T2M, which reports FID 0.265 on the HumanML3D-E subset with at least four events, versus 0.418 for MoMask (Event-T2M). For broader context, Motion-R1 reports FID 0.53 on BABEL (Motion-R1). Our loop on an MDM base will likely not beat 0.265 on FID alone. So the claim against Event-T2M is twofold: repair on top of Event-T2M should improve its event completion while keeping FID within noise of 0.265, and repair on top of MDM should close a measured fraction of the MDM-to-Event-T2M gap.

Ablation of the key ingredient. We keep the checker and the budget fixed and replace localized masks with whole-sequence re-noising at the same noise level. This isolates localization from "more denoising." A second ablation replaces every language-model role with non-agentic counterparts: a fixed segmentation for the planner and the retrieval score for the checker.

Why not just…?

Why not just use best-of-N with a good motion-text score? It is simple and parallel, and with an accurate scorer it might suffice when the base model already gets most prompts right. The evidence against it in sequential settings is the 22.7% vs 0.7% gap in Temporal Backtracking Search. The reason is structural: with four or more events, the chance that one full sample gets every event right shrinks multiplicatively, while repair fixes events one at a time. Our scaling curves of event completion versus function evaluations, split by event count, will show whether this holds for motion or whether best-of-N keeps pace.

Why not just restart from the last good prefix, as Temporal Backtracking Search does? This is the real threat, and it may be nearly as good when failures cluster late. It loses by construction when a middle event fails and later events were correct, because it regenerates and can break them. We will stratify results by failure position. If prefix restart matches us on mid-sequence failures, the bidirectional mask is not earning its place and we will say so.

Why not just use gradient guidance for physical constraints? Constraint-gradient methods such as Guided Motion Diffusion and the Langevin view in Inference-time scaling of diffusion models through classical search likely fix foot sliding and penetration better and more cheaply than any agent. We therefore do not claim kinematic repair as a contribution. We include guidance as a compatible component and focus the claim on semantic event failures, which have no differentiable target.

Why not just have one multimodal model reason about the whole motion and fix it? A single long-reasoning critic is exactly the regime where More Thinking, Less Seeing? observes attention drifting away from visual evidence. We run this arm explicitly: one vision-language model watches the full render, reasons at length, and outputs a mask. We compare its mask localization accuracy against our per-window short queries.

Why not just drop the agents and use retrieval scores? This may work as well, and the second ablation tests it. If retrieval scores localize failures as accurately as the vision-language checker, the final method uses no language-model checker.

Proposed Experiments

Data and evaluation sets. The main evaluation is the HumanML3D-E subsets stratified by event count (Event-T2M), with standard HumanML3D metrics: FID, R-Precision, MM-Dist, diversity, and foot skating (Generating Diverse and Natural 3D Human Motions from Text). Because those metrics do not directly score whether each event happened in order, we add a second set built from BABEL. BABEL's frame-level action labels let us compose prompts with known event order and spans.

Event completion. We score event completion two ways. One is per-segment retrieval against the gold event text, using a held-out retrieval model distinct from the checker to avoid scoring with the optimized verifier. The other is a human study of event presence and order. We do not rely on an action classifier alone, since BABEL reports only 44.87% Top-1 for its supervised baseline.

Decisive check 1. At matched function evaluations, the localized loop must beat best-of-N and prefix restart on event completion for prompts with four or more events, and the gap must widen with event count.

Decisive check 2. The localized-mask ablation must underperform full re-noising's opposite. That is, if whole-sequence re-noising at equal budget matches us, localization is not the cause.

Decisive check 3. On mid-sequence failures, bidirectional masks must preserve already-correct later events, measured as the retention rate of previously passing events, better than prefix restart.

Secondary results. We report which repair operator wins, stochastic re-noising or deterministic re-conditioning. We also report how repair gains fall as checker precision is degraded with controlled label noise, which gives a direct reading of how much the method depends on its verifier.