Seam-Repaired Body-Part Recombination for Low-Data Motion Forecasting

Models that forecast 3D human motion, meaning they predict where every joint of a skeleton will be over the next second from a short observed clip, fail most visibly on combinations they have never seen. Examples are an arm gesture performed while walking, or a turning torso over a shuffling gait. These failures are about coupling: how the timing and geometry of one body part depend on another. This proposal generates synthetic training motions that deliberately create new part-to-part couplings. We cut the arms, legs, and torso from different real clips and align their gait timing. A diffusion model, which is a generator that learns to turn random noise into realistic samples by repeated denoising, then repairs only the joints and frames near the cut, so the spliced motion becomes physically coherent. The generator's job is kept deliberately small. It fills in a seam and does not invent whole motions from scratch.

The claim is that, in the small-data regime where generative augmentation is known to pay off, recombining real part trajectories and repairing the seams beats both cheap heuristics and whole-sequence diffusion sampling at equal synthetic budget. We further claim the benefit comes from new couplings rather than new poses. If that holds, it says what kind of synthetic variation a forecaster actually needs. It also offers a way to grow small skeletal cohorts without the hallucination drift that makes blind generation harmful.

Why this, and why now

The forecasting literature has converged on the finding that spatial structure and temporal dynamics should be handled separately. 3D Human Motion Prediction via the Decoupled Spatiotemporal Clue uses isolated spatial, temporal, and cross attention branches and argues that monolithic coupled networks blur individual joint trajectories. Spatio-Temporal Branching for Motion Prediction using Motion Increments splits frame differences into spatial and temporal graph branches. Skeleton Graph Scattering Networks decomposes the body into parts and fuses them across parts. All of this work motivates our proposal by showing that part-level coupling is where forecasters struggle. None of it changes the training data. Each of these papers treats the coupling statistics present in the dataset as fixed.

The generative augmentation side has two partial bridges. SGAR swaps body-part segments across sequences ("Part-Mix") to break spurious co-occurrences in motion–text retrieval. It works on embeddings, however, and its authors note that it has no temporal alignment, so a raw swap produces kinematic discontinuities. We borrow the part decomposition and fix exactly the two things SGAR leaves open: we produce continuous joint trajectories, and we align the parts in time. PoseAugment generates whole pose sequences with an autoregressive variational autoencoder and cleans them with a physics solver based on quadratic programming. That paper constrains our design in two ways. First, it shows that unconstrained generation is badly jittery: raw generator output has a third-derivative jitter of 5.15, against 2.11 in real data, and PoseAugment brings this down to 2.32. Second, it shows that a physics solver is expensive. We avoid both costs by keeping most of each synthetic sample as untouched real motion.

Two further results set the operating regime. Toward Understanding Generative Data Augmentation argues through algorithmic stability that synthetic data helps when the real set is small. It also shows that synthetic data can hurt when data is abundant or the generator is imperfect: GAN samples lowered augmented CIFAR-10 accuracy from 94.4% to about 92.6%. Active Generation Network shows that filtering generated skeleton sequences by downstream uncertainty added 4.76% action-recognition accuracy over unfiltered generation. We therefore target limited-subject training rather than full Human3.6M, and we include a filtering stage.

Core Claim

A forecaster trained on a small set of real clips improves more from synthetic clips that recombine real part trajectories with aligned timing and repaired seams than from the same number of clips sampled whole from a diffusion model trained on the same data. The advantage should be largest on test motions whose part combinations are absent from training. It should shrink toward zero as the real training set grows, as Zheng et al. predict.

Research Program

The pipeline has four stages.

Decompose. Each training clip is split into the five parts SGAR uses: the two arms, the two legs, and the torso with the root.

Align. For each part we estimate a local phase and dominant frequency from its trajectory. We use the channel-wise Fourier decomposition of FunPhase where the motion is periodic, and a cross-correlation fallback where it is not, because FunPhase reports that its periodic model over-smooths aperiodic movements. A donor part from another clip is time-warped so its phase matches the host clip's lower-body cycle. The part's length in bones is then retargeted to the host skeleton.

Repair. The spliced sequence is passed to a masked diffusion model that works on DCT coefficients, the Discrete Cosine Transform frequency representation that most forecasters in this area use. The model follows HumanMAC, which frames motion completion as inpainting in the DCT domain. In our version, the mask frees only the seam joints (shoulders, hips, spine, and neck) plus a short window of frames around each warp boundary. All donor and host joints away from the seam stay fixed. Training this inpainter is cheap because it only has to learn local consistency. To keep dataset construction fast, we will test the single-step distilled variant from SPARD, which reports a 15–18x sampling speedup, against the full multi-step sampler.

Filter. Repaired clips are rejected if they fail simple checks: bone-length deviation, foot sliding, and jitter measured against the real-data statistics that PoseAugment reports. The survivors are scored AGN-style by how much an ensemble of forecasters trained on the real data disagrees on them. High-disagreement clips are kept, and synthetic data is capped at a 1:1 ratio with real data. Generative Spatiotemporal Data Augmentation found 1:1 to be a safe ceiling for video detection, and we will sweep the ratio rather than assume it transfers.

The analysis layer measures what actually changed. For each synthetic set we compute cross-part coupling statistics, meaning correlations between part velocities, and compare them to those in the real training and test sets. This tests whether gains track coverage of new couplings or merely coverage of new individual poses.

Baselines

The naive reference point is the downstream forecaster trained on the limited real subset with no augmentation. The main backbone is the decoupled-attention model from 3D Human Motion Prediction via the Decoupled Spatiotemporal Clue. We chose it because it is small (a reported 2M parameters) and fast (a reported 13.9 ms), which makes many seeded training runs affordable. We will repeat the key comparisons with a second backbone so the conclusions are not tied to one architecture. Every method uses the same real subset, the same backbone and optimizer, the same number of training steps, and the same hyperparameter search budget, with three or more seeds per cell.

The heuristic baseline applies random 3D rotations, left–right mirroring, and Gaussian joint noise, each with its magnitude tuned on validation. This is what a practitioner would try first.

The strongest existing method that uses no external synthetic data is AuxFormer. It adds denoising and masked-coordinate-recovery auxiliary tasks during training, and it reports outperforming prior methods by 7.2% MPJPE (mean per-joint position error) on Human3.6M and by 9.4% on 3DPW when trained on the full data. It is the strongest competitor because it targets the same goal, regularizing a forecaster through corruption, at no data-generation cost. We will run it at its reported best settings, a 50% mask ratio and a 30% noise ratio, and also give it the same tuning budget on our subsets, since those settings were chosen at full data. For absolute scale on full Human3.6M, Learning Progressive Joint Propagation reports an average MPJPE of 67.6 mm at 1000 ms. We found no reported result on Lacuna for any method on subject-limited Human3.6M training, so all low-data numbers will be produced by us.

The strongest generative baseline is whole-sequence augmentation. We train a HumanMAC-style DCT diffusion model on the same real subset and sample full clips, using the same count, the same filter, and the same ratio as our method. This isolates what the recombination step contributes.

Two ablations isolate the key ingredients. Raw Part-Mix, a trajectory-level version of SGAR with neither alignment nor repair, tests whether cut-and-paste alone is enough. Aligned Part-Mix without repair tests whether the diffusion seam repair matters.

Beating the field means a statistically reliable MPJPE reduction at 400 ms and 1000 ms over both AuxFormer and whole-sequence diffusion in the low-data settings, without worse physics metrics. The compositional split described below is the decisive test of that claim.

Why not just…?

Why not just use heuristic augmentation? Rotation and mirroring leave every part-to-part coupling unchanged, so by construction they cannot supply new combinations. Joint noise does perturb couplings, but it breaks bone rigidity. PoseAugment reports that noise-style augmentation produces physically impossible inputs, and AuxFormer shows that forecasters trained without structured corruption degrade sharply under noisy inputs. The heuristic baseline stays in every table, and the compositional split will show whether it closes any of the gap.

Why not just use AuxFormer's in-situ masking and denoising? This is a real threat, especially at moderate data sizes, because it adds robustness at no generation cost. Its limitation is that every training example is still a corrupted version of a real clip, so it cannot show the model an arm pattern paired with a gait it never co-occurred with. If AuxFormer matches us on the compositional split, our claim fails. We will also test the two methods combined, since they are not mutually exclusive.

Why not just splice parts without a generator? Raw Part-Mix is the cheapest version of our idea, and if it works nearly as well, the diffusion repair is unnecessary. SGAR's own authors flag temporal discontinuity as its weakness. We still treat the question as empirical, running both the raw and the aligned-only ablations. If aligned splicing alone matches the full pipeline, we will report the simpler method as the contribution and drop the generator.

Why not just sample whole motions from a diffusion model? Diffusion samples outperformed GAN samples in Zheng et al.'s experiments, so this is a credible alternative. But a generator trained on few clips mostly reproduces the couplings it saw, and PoseAugment's jitter figures show how far whole-sequence generation can drift physically. The equal-budget comparison and the coupling-coverage analysis will show which effect dominates.

Why not just pretrain on AMASS? In the standard skeleton setting this could erase the benefit of any augmentation. That is why we also run every method on top of an AMASS-pretrained backbone. If the gain vanishes there, the method's value is limited to settings without a large matching corpus, such as unusual skeletons or small clinical cohorts like the 54-patient gait data in GaitForeMer. We would state that scope plainly.

Proposed Experiments

The primary benchmark is Human3.6M forecasting under subject-limited training: one subject, two subjects, and the full five standard training subjects, evaluated with MPJPE at 80, 160, 400, and 1000 ms on the standard test subject. CMU Mocap provides a second dataset, and 3DPW is used for transfer evaluation.

We will also build a compositional split. Part trajectories are clustered separately for the arms and the legs, and every clip whose (arm cluster, leg cluster) pair falls in a held-out set is withheld from training. The test set then contains part combinations absent from training. This is where the core claim lives. Our method should beat whole-sequence diffusion and AuxFormer by a clearly larger margin here than on the standard split.

The physics and quality metrics are foot sliding, third-derivative jitter, bone-length error, and a seam discontinuity score, each computed for the synthetic sets and for the forecasts. Dataset-construction cost will be reported in GPU-hours for multi-step versus single-step repair.

Three results would be decisive against the proposal:

  1. Aligned Part-Mix without repair matching the full pipeline, which would remove the need for a generator.
  2. AuxFormer matching our method on the compositional split, which would mean in-situ corruption already covers new couplings.
  3. Gains that fail to shrink with more real data. This would contradict the stability account of Zheng et al. and suggest the improvement comes from something other than synthetic coverage, for example extra training steps, which the matched-step design is meant to rule out.

The ratio sweep and the coupling-coverage correlation complete the picture by showing how much synthetic data to add and why it helps.