Searching Scene-Aware Motion Models on the Cheap, Scene-Free Task

Models that forecast human motion come in two kinds. Environment-agnostic models see only the past skeleton, a sequence of 3D joint positions, and extrapolate it as if the person moved through empty space. Scene-grounded models also read the room, as a point cloud or voxel grid, and they avoid artifacts like feet sinking through floors or hands passing through tables. The scene-grounded models are far more expensive to train, because every candidate carries a heavy scene encoder. That cost makes automated architecture search, where many candidate networks are trained and compared, especially painful in exactly the setting where hand-designed models are known to be too slow for robots. This note proposes treating the scene as a fidelity knob. Candidate motion backbones are ranked on the cheap scene-free version of the task. Only a small, separately searched interface that injects scene information is then paid for at full cost.

The claim is empirical and can fail. On the same data, the ranking of skeletal backbones with the scene removed should predict their ranking once a scene branch is attached, but only at short prediction horizons. At long horizons the scene takes over the job the backbone's internal priors used to do, which is keeping motion from drifting toward a frozen average pose, so long-horizon scene-free scores should mislead. If this holds, search can be split into two stages. The first ranks backbones cheaply using short-horizon, scene-free scores. The second grows each chosen, already-trained backbone into a scene-aware model with function-preserving insertions, and searches only where and how scene features enter. The main deliverable is the measurement of when scene-free rankings transfer. The search method is justified only if it beats the simple alternatives below at matched compute.

Core Claim and How It Differs From Prior Work

The closest existing work is TrajGNAS. It runs differentiable weight-sharing search over graph operators for 2D multi-agent road trajectories, adding a lane-compliance loss and a FLOPs penalty. We differ on three axes. Our target is full-body 3D skeletal motion in indoor scenes. We make no use of a weight-sharing super-network. Our cost lever is a fidelity axis, "with or without environment," that has no analogue in road-trajectory search.

The hand-designed scene-aware models motivate the problem. GIMO fuses PointNet++ scene features, gaze and motion through a bidirectional transformer. TRUMANS conditions an autoregressive diffusion model on local voxel grids through a Vision Transformer. GAP3DS conditions a diffusion generator on gaze-derived affordances. All three fix their fusion design by hand and pay heavily at inference.

Context-Aware Human Motion Prediction supplies the key hypothesis. It shows that scene-blind models drift toward a mean pose over long horizons and that adding object context fixes this. It also shows that human–object influence is asymmetric: the table shapes the reach, but the reach does not move the table. GIMO argues the opposite and uses symmetric bidirectional fusion. Our fusion search makes the choice between the two an explicit, measured decision.

On the architecture-search side, Rethinking Performance Estimation in NAS supplies the methodology. It treats training epochs, image size and depth as budget dimensions and asks which of them can be cut without destroying rank correlation. We add "environment present" and "forecast horizon" as new dimensions of that kind. Efficient Architecture Search by Network Transformation supplies the growth mechanism. It modifies trained networks with function-preserving edits, so candidates need only short fine-tuning.

Two lines of work act as constraints. Zero-Cost Proxies for Lightweight NAS scores untrained networks with gradient-saliency statistics at initialization. NAS-Bench-Suite-Zero and EZNAS report that such scores degrade on non-standard topologies, favour large models, and are unreliable within the top decile. These findings make training-free scoring an important baseline, not an assumed solution. The same zero-cost paper also warns that FLOP reductions translate poorly into wall-clock savings. For that reason every cost claim here is measured in GPU-hours on real hardware, not in FLOPs.

Research Program

Building the paired benchmark. First we build a paired benchmark that does not yet exist. The backbone search space follows the spatio-temporal graph networks used by HVIS, Dynamic Dense GCN and Learning Semantic Latent Directions. These are graph networks over the skeleton, meaning message passing along bones combined with temporal processing. The searchable choices are:

  • whether spatial and temporal adjacency are coupled or decoupled (the choice HVIS argues matters);
  • the number of hierarchy levels (joints, limbs, whole body);
  • whether the sequence is encoded in the frequency domain with a discrete cosine transform;
  • depth and width.

We sample on the order of two hundred backbones. Each is trained to convergence twice on TRUMANS and on GIMO: once scene-free, with scene inputs deleted, and once with a fixed reference scene branch, a small voxel encoder feeding cross-attention. Using the same data for both versions isolates the effect of the environment from any dataset shift.

A secondary, cross-dataset condition trains the scene-free version on Human3.6M after mapping to a shared joint subset. It asks whether publicly abundant scene-free motion data can serve as the cheap stage. Seeds are repeated, and seed-to-seed noise is reduced with training-log covariates as in Can Training Logs Make Model Comparisons More Precise?. This matters because GIMO has only 217 trajectories, and ranking noise there is a real concern.

The central measurement. For forecast horizons $h$ (the scene-free scoring horizon) and $H$ (the scene-grounded evaluation horizon), we compute

$$\tau(h, H) = \text{KendallTau}\big(\text{score}_{\text{free}}(a; h),\ \text{score}_{\text{grounded}}(a; H)\big)$$

over backbones $a$, where each score is mean per-joint position error at the given horizon. We report $\tau$ both over the whole population and within the top decile. The hypothesis predicts that $\tau$ is high when $h$ is short (around 0.5 s), even for long $H$. It also predicts that long-horizon scene-free scores ($h$ near 2 s) transfer worse, because they reward drift-suppression machinery that becomes redundant once scene context is present.

Growing the scene-aware model. The second stage takes the top-$k$ backbones, already trained scene-free, and grows them. We insert scene cross-attention blocks whose output projections are initialized to zero, so the grown network initially reproduces the scene-free predictions exactly. This is the function-preserving property that EAS exploits. The fusion search space covers:

  • which layers receive scene input;
  • whether attention runs one way (motion reads scene) or both ways;
  • scene-encoder width;
  • voxel resolution.

Each candidate is evaluated after a short, fixed fine-tuning budget. The search objective combines three terms: long-horizon joint error, the scene-penetration rate as defined for GIMO in GAP3DS, and measured latency on the deployment device.

Baselines

Naive reference. The simplest reference point is the best published-style scene-free backbone with no scene input at all. It sets the floor for what the scene buys on each metric.

Scene model without search. The second is the same backbone with a single fixed fusion design, bidirectional cross-attention at every layer in the style of GIMO, trained from scratch. It stands for current practice.

Strongest external methods. On GIMO, the strongest reported numbers in the file are:

  • GIMO's own model: 237.8 mean per-joint error, 0.245 translation and 0.579 orientation destination error on known scenes, per GIMO;
  • GAP3DS: a 0.95% penetration error, against 4.59% for SIF3D, per GAP3DS.

Both models use gaze, which our models do not. For the comparison to be fair, we either add the same gaze input to our fusion search space or compare against GIMO's gaze-free variant, which reports 262.1 mean per-joint error. For TRUMANS, the file reports only a sampling speed (1.6 s of motion in 0.7 s on an A800) and a user study. We found no reported forecasting-error result on Lacuna for TRUMANS, so we will retrain the reference model ourselves under our splits.

Strongest search competitor. The strongest competitor on the actual question is a TrajGNAS-style method: differentiable weight-sharing search over the joint backbone-plus-fusion space, with a FLOPs penalty and our penetration term substituted for its lane loss. All search methods get the same GPU-hour budget, measured wall-clock. They share one search space and one final protocol, in which the top five architectures each method selects are retrained from scratch with three seeds. The same tuning budget is allowed for each method's own hyperparameters.

Ablations. Two ablations isolate our ingredients:

  • scene-free ranking at a long horizon instead of a short one, which tests the horizon hypothesis;
  • growth with ordinary random initialization of the inserted blocks instead of zero initialization, which tests whether function preservation matters.

What counts as winning. Beating the strongest competitor means one of two outcomes at equal search budget. Either the factored method reaches a latency–error–penetration frontier that dominates the competitor's, or it matches that frontier at no more than a fifth of the GPU-hours.

Why not just…?

Why not just bolt a zero-initialized adapter onto every layer of a good pretrained backbone and skip search entirely? This is the most serious threat. Function-preserving growth with a fixed, generous adapter is exactly this baseline, and it may already be good enough. We run it explicitly, with fusion at every layer and bidirectional attention. The paper's claim then narrows to "search pays off" only if the searched interfaces are clearly faster at equal penetration and error. The asymmetry evidence in Context-Aware Human Motion Prediction suggests that pruning one attention direction and most insertion sites should be nearly free. If the all-layer adapter matches the searched frontier, we will report that. The transfer measurement, meaning that backbones can be chosen scene-free, remains useful on its own.

Why not just score candidates with off-the-shelf zero-cost proxies? They are orders of magnitude cheaper, so they are included as a baseline. We apply the NAS-Bench-Suite-Zero proxy set directly to the full scene-grounded candidates and correlate the scores with converged performance. There are grounds for doubt. Zero-Cost Proxies for Lightweight NAS found that activation-correlation proxies collapse on heterogeneous architectures. NAS-Bench-Suite-Zero found that proxies track parameter count, which is the wrong incentive for a latency-bound robot. And no initialization statistic can see whether a network will learn to avoid penetrating scene geometry. The paired benchmark lets us settle this with numbers instead of argument.

Why not just train every full scene model for a few epochs? Reduced training is the obvious cheap fidelity. Rethinking Performance Estimation in NAS shows that arbitrarily chosen short budgets can scramble rankings, and the GPU-gap result warns that the savings may be smaller than they look. The real test, though, is our own comparison at a matched budget. That comparison sets short full-model training against full scene-free training of the backbone, with epoch counts picked by the same importance-pruning procedure for both. If short full training transfers better per GPU-hour, the scene-as-fidelity idea is wrong.

Decisive Checks

Three results decide the proposal:

  1. Transfer at short horizons. On TRUMANS and GIMO, top-decile $\tau(0.5\text{ s}, 2\text{ s})$ must clearly exceed $\tau(2\text{ s}, 2\text{ s})$. If it does not, the horizon hypothesis fails.
  2. Frontier at matched cost. At matched measured GPU-hours, the factored search must dominate the TrajGNAS-style joint search, the zero-cost-proxy search and the short-training search. Domination is judged on the three-way frontier of long-horizon mean per-joint error, penetration rate and on-device latency, with final models retrained from scratch.
  3. Search beats a generous fixed adapter. Searched fusion must beat the every-layer zero-initialized adapter on latency without losing penetration. If only the first check holds, we publish a validated recipe for choosing backbones scene-free. If none hold, we publish the paired benchmark and the negative result, both of which the two fields currently lack.