Cheap, Rank-Faithful Search for Motion Control Branches When Data Is Scarce

Adding a second control signal, such as music or speech, to a text-to-motion generator is usually done by copying the whole pretrained network into a trainable "control branch." The original stays frozen, and the two halves are joined by small linear layers initialized to zero. This keeps the text skill intact, but it doubles the model's size, and nobody has checked whether every layer actually needs its own copy. Searching for a leaner branch looks like a standard architecture-search problem. Two features of 3D motion make it unusual. First, training data is tiny: tri-modal text, audio and motion data barely exists, and music-dance sets are small. Second, the evaluation sets are just as small, so the metrics used to compare candidates may be noisier than the differences between them.

We propose to build the first small, fully trained reference set of control-branch architectures for music-conditioned motion. We will use it to measure which cheap evaluation methods actually preserve the ranking of architectures under low data. Only then will we run a search with the method that survives. The scientific question is whether, in this regime, the binding constraint on architecture search is training compute or evaluation noise. The practical question is whether a searched branch that is much smaller than a full copy can match the full copy's music alignment.

Core Claim

MCM freezes a pretrained text-to-motion diffusion transformer and duplicates it into a trainable branch for Jukebox audio features, joined by zero-initialized bridges. It reports a Beat Alignment Score of 0.275 on AIST++. The design forces identical depth, width and temporal resolution on the audio path, even though music changes on sub-second beats while text describes multi-second intent.

Adaptable Adapters makes the opposite design choice. It learns, per layer, whether to insert an adapter at all. In BERT with 100 training examples, it scored 51.28 average GLUE against 44.87 for uniform adapters, using about 13 of 24 layers. That result motivates our search space, but it comes from classification with plentiful evaluation data.

Our claim has two parts. Under motion-scale data, cheap evaluation methods designed for image classification will rank control branches poorly, and we can say in advance why. The forward-only zero-cost scores collapse by construction: with zero-initialized bridges, every candidate produces exactly the same output at initialization. Short-training estimates are limited by how reliable the full metric itself is on a small test set. We will show that a budgeted estimator built for motion recovers most of the ranking that the evaluation noise allows. It does this by scoring single-step clean-motion predictions from noised held-out clips on beat alignment and foot-contact error, rather than sampling full sequences. With that estimator, a search over bridge placement, branch width and audio frame rate should find branches far smaller than a full copy at equal beat alignment. We also expect the best branch to shrink as training data shrinks.

Research Program

Frozen backbone. The text model stays frozen throughout. We will use whichever of MWNet, MDM or MLD has usable released weights. MDM is convenient because it predicts the clean motion at every step and therefore supports kinematic losses.

Search space. The search space covers only the control interface. For each backbone layer, we choose whether a bridge exists, the control block's width as a fraction of the main width (zero, a quarter, a half, or full), and whether the audio stream enters at the full frame rate or pooled to beat-level resolution. A coarse operator choice, attention versus a linear-time state-space scan in the spirit of Motion Mamba, is a later extension. The full MCM copy is one point in this space, so the search can recover it.

Stage one: the reference set. We fully train roughly 50 architectures on AIST++ with three seeds each. The architectures come from stratified sampling over parameter count, plus the hand-designed heuristics described under Baselines. At an assumed cost of around one GPU-day per run, this is on the order of 150 GPU-days. It is the expensive, one-time part of the project and will be released as a small tabular benchmark. From the seed variance and bootstrap resampling of the test set, we compute the reliability of each full metric. That reliability caps the rank correlation any cheap estimator can reach, and we report every estimator against that cap, not against 1. The concern here, that small evaluation sets make model comparisons imprecise, is related to the question posed by Can Training Logs Make Model Comparisons More Precise?, and we test training-log features as one estimator family.

Stage two: estimator comparison. We compare the following cheap estimators on the reference set using Kendall's tau:

  • Zero-cost scores. SynFlow, NASWOT, and an expression evolved in the manner of EZNAS, which reached τ = 0.65 on NAS-Bench-201 CIFAR-10. Gradient-based scores are not degenerate under zero-init; activation-based ones are.
  • Budgeted short training. Short runs with budgets tuned as in Budgeted Performance Estimation, which cut search from 3,150 to 0.33 GPU days on CIFAR-10.
  • Teacher divergence. Divergence from the full-copy model, analogous to the relative FID used by Flexiffusion.
  • Our single-step estimator. Beat alignment and foot-contact error computed on single-step predictions after a short budget.

We report correlation over all architectures and also within the top 10%. EZNAS documents that zero-cost proxies break down in that top tier, and it is exactly where a search lives.

Stage three: the search. We run evolutionary search with the winning estimator at three training-data fractions of AIST++ (full, half, quarter) to test whether the optimal branch shrinks. We then transfer the search procedure, not the found architecture, to BEAT co-speech gestures.

Baselines

Full-copy branch. This is MCM's dual-branch design, retrained by us on the same AIST++ splits and with the same frozen backbone, so that only the branch differs. It is the strongest existing method on this question. The file's only reported number for it is a Beat Alignment Score of 0.275 on AIST++. We found no reported result on Lacuna for MCM's motion FID or latency on AIST++, so we will measure those ourselves. Beating it means matching its beat alignment, within the seed confidence interval from our reference set, at no more than half its control parameters. Alternatively, it means exceeding it at equal size.

Naive reference points. The do-nothing reference is the frozen text model with no audio, which fixes the floor for beat alignment. The second naive point is random search over the same space with the same GPU budget and the same final-training protocol.

Hand-designed heuristics. These are about six sparse branches a careful engineer would try: bridges only in the deeper half (the pattern Adaptable Adapters' gating found), every other layer, uniform half width, uniform quarter width, and beat-pooled audio everywhere. All are trained identically. This is the baseline most likely to tie the search.

One-shot differentiable gating. This is a single supernetwork with per-layer gates trained jointly, in the style of Adaptable Adapters and DARTS. It gets the same total compute as our search. We evaluate its derived architecture after retraining from scratch, and we also score how well its internal gate values rank the reference set.

Ablation. The key ablation replaces our motion-specific single-step estimator with plain held-out denoising loss at the same budget. This isolates whether the beat and contact terms carry the ranking signal.

Why not just…?

Why not just keep MCM's full copy? It works, and if no smaller branch matches it, that is the answer. But Adaptable Adapters found that about half of the adapter layers were unnecessary in low-data transfer. MCM forces full capacity at every depth and at the frame rate of the audio. The reference set directly measures how beat alignment falls off with branch size, so this question is settled empirically rather than argued.

Why not just hand-pick a sparse branch, such as deep layers only? This is a real threat. If the best heuristic lands within seed noise of the searched branch, the honest conclusion is that search is unnecessary here. The contribution then shrinks to the benchmark, the estimator-reliability analysis, and evidence that the heuristic is near-optimal. The data-fraction experiment gives search its best chance to matter: if the optimal size shifts with data, a fixed heuristic cannot track it.

Why not just train differentiable gates once? It is the cheapest option. However, Rethinking Performance Estimation documents rank inversion in weight-sharing supernets, and zero-initialized bridges inside a frozen network may make co-adaptation worse. We measure this directly as the Kendall tau between the supernetwork's gates and fully trained performance. If that correlation is high, we adopt gating and drop evolution.

Why not just use a zero-cost proxy? Activation-based scores see identical outputs for every candidate at initialization because of the zero bridges, so they are uninformative by construction. Gradient-based scores may survive, and they are included and scored against the reference set.

Why not just use fewer sampling steps? Faster samplers cut the number of denoising steps but leave the per-step cost of a doubled network unchanged. Flexiffusion reports structural gains, 66G MACs against DeepCache's 130G, on top of sampling schedules. The two approaches combine, and we report latency both ways.

Proposed Experiments and Decisive Checks

All motion experiments use AIST++. We report:

  • Beat Alignment Score.
  • Motion-quality FID and diversity on held-out music.
  • Foot-skating rate.
  • Control-branch parameters and per-sequence latency.

To confirm text adherence survives joint conditioning, we report HumanML3D R-Precision with pseudo-text prompts, as MCM does. BEAT serves as the transfer test.

Three results would be decisive.

Estimator ranking. The first is whether any cheap estimator reaches at least half the metric's reliability cap in top-10% Kendall tau. If none does, search under motion-scale data is not yet viable, and that negative result is itself worth publishing.

Searched versus heuristic branches. The second is whether the searched branch beats the best heuristic and MCM's full copy on the size-at-equal-beat-alignment criterion.

Data scale. The third is whether the selected branch's capacity shrinks monotonically from full to quarter data, which would show that the right architecture depends on how scarce the data is.