Body-Anchored Stitching for Video Depth and Human Motion

Depth estimators run on video tend to flicker: each frame, or each short chunk of frames, gets its own arbitrary scale and offset, so the reconstructed world swells and shrinks over time. A recent fix converts a video diffusion model (a generator trained to turn noise into video, here run once at a fixed noise level and used as a plain predictor) into a deterministic depth estimator. Because the predictor no longer samples at random, overlapping chunks can be stitched by solving a small least-squares problem for one scale and one shift per chunk. That fix assumes the scene is effectively rigid. People moving through the frame are exactly where the assumption is most likely to fail, and they are often the reason we want depth in the first place. Human motion estimators have the opposite weakness. They know the shape of a body, but they drift in depth and in world position, because they borrow depth from an external model that flickers.

We propose to make each side fix the other's weakness without any iterative test-time optimization. A single deterministic pass of a video-diffusion backbone predicts dense depth and per-frame body meshes from the SMPL body model. Overlapping chunks are then stitched with a closed-form solve. In that solve, each body part gets its own scale and shift, tied to the background, and the predicted body mesh acts as a metric ruler that fixes the scale of the whole scene. The stitched depth then fixes the body's root depth in closed form. The bet is that this one linear solve per chunk recovers what slow gradient-based fitting recovers, at a small fraction of the cost.

Core Claim

DVD turns Wan2.1-1.3B into a single-pass video depth regressor by fixing the diffusion timestep. It argues that with sampling noise removed, disagreements between overlapping windows are affine, and it stitches 1,500+ frame sequences with one global least-squares alignment. It reports AbsRel 5.5 on ScanNet and 6.7 on KITTI, plus boundary F1 of 0.259 versus 0.210 for Video Depth Anything on ScanNet. Its evaluation is on static and driving scenes.

Our claim has two parts. First, on videos dominated by moving people, the window-to-window disagreement is affine per segment but not globally: the person's depth relative to the background shifts between windows in a way one $(s,t)$ pair cannot absorb. Second, a predicted body mesh supplies exactly the extra constraints needed to resolve this cheaply. It labels which pixels belong to which part, and its known metric size pins down absolute scale.

If the first part is false, meaning the residual of DVD's global fit inside human masks is no larger than on the background, the stitching contribution disappears. The decisive first experiment tests exactly this. The second part, closed-form depth–pose coupling replacing iterative fitting, is the latency contribution and stands on its own.

Research Program

The backbone follows DVD's recipe: a video diffusion transformer run at a fixed timestep, producing per-window depth. We add a light head that reads the backbone's features and regresses per-frame SMPL pose, shape and camera-frame translation. The head borrows its design from DPMesh, which repurposes the Stable Diffusion denoiser as a one-step feature extractor for occluded single-image mesh recovery and conditions it on 2D keypoint heatmaps. We extend DPMesh from single images to video windows. The video backbone's temporal attention gives occluded limbs context from neighbouring frames that DPMesh cannot see. This is where the generative prior earns its place: as structure inside features, not as a sampler.

Stitching works as follows. Take two overlapping windows $k$ and $k+1$, with depths $D_k$ and $D_{k+1}$ on their shared frames. Split the shared pixels into a background segment $b$ and body-part segments $p$, using the rendered SMPL part labels. Let $z_p$ be the depth of part $p$ rendered from the predicted mesh. We solve

$$ \min_{\{s_g,t_g\}} \sum_{g\in\{b\}\cup P}\sum_{x\in g} w(x)\big(s_g D_{k+1}(x)+t_g-D_k(x)\big)^2 + \lambda\sum_{p\in P}\big\|(s_p,t_p)-(s_b,t_b)\big\|^2 + \mu\sum_{p\in P}\sum_{x\in p}\big(s_p D_{k+1}(x)+t_p-z_p(x)\big)^2 . $$

The symbols are defined as follows:

  • $w(x)$ is a confidence weight that down-weights depth edges and pixels the mask marks as uncertain.
  • $\lambda$ keeps part-level corrections near the background correction, so a part cannot drift freely.
  • $\mu$ ties body pixels to the mesh's metric depth. This term lets the person serve as a ruler that also sets the background's absolute scale.

The problem is a small quadratic in a few dozen unknowns, so it has an exact linear solution. Once depth is stitched, we update each frame's root translation by aligning rendered mesh depth to the stitched depth inside visible body pixels. That is a one-dimensional weighted least-squares problem per frame.

Running in a streaming mode adds delay of one window rather than one frame. We say so plainly and compare against OnlineHMR on both accuracy and wall-clock latency.

Baselines

The naive reference applies DVD per window, uses DVD's global affine stitching, and runs a per-frame body regressor with no depth coupling. It shows how much is gained by doing anything at all.

The strongest depth competitor is DVD's global stitching made robust: refit the global affine with human pixels masked out as outliers. This is what a careful practitioner would do first. It shares our backbone, windows and overlap, so any difference comes from the alignment model alone. Beating it means lower AbsRel and lower frame-to-frame depth change inside human masks at equal background accuracy.

The strongest pose competitors are three methods:

  • OnlineHMR is causal, uses a key-value cache, and takes depth from MoGe-V2. It reports 3.3 FPS at 0.30 s latency and better WA-MPJPE than TRACE and Human3R on EMDB-2; no reported scalar WA-MPJPE was found on Lacuna.
  • WHAM combines keypoint lifting, SLAM angular velocity and contact-aware refinement, with roughly 9 FPS online. It claims lower root translation error on EMDB, but no reported RTE value was found on Lacuna.
  • Depth-Guided HMR fuses Depth Anything v2 features with RGB and calibrates bone lengths from depth. It reports MPJPE 69.31 mm on 3DPW and is the closest existing depth–pose coupling.

For occlusion, the reference number is DPMesh's 70.9 mm MPJPE on 3DPW-OC. Beating these means matching their MPJPE while reducing acceleration error and world-frame drift.

All baselines are run on identical test splits with their released settings. Methods we retrain get the same training data (3DPW train, Human3.6M) and the same tuning budget as ours.

Two ablations isolate the key ingredients. The first removes the mesh-ruler term ($\mu=0$) and the part-wise terms in turn. The second replaces our closed-form root and depth solve with gradient-based fitting of SMPL translation and scale to the stitched depth, run for 10, 50 and 200 Adam iterations. The proposal succeeds on latency if the closed-form version matches the 200-iteration version within noise at the cost of one forward pass.

Why not just…?

Why not just mask humans out of DVD's global affine fit? This fixes the background but leaves the human's own depth unaligned across windows. The human is where errors in augmented-reality occlusion and in world-grounded motion actually show up. The robust-global baseline measures directly how much human-region error remains. If that error is already small, we drop the part-wise terms and keep only the ruler coupling.

Why not just swap the depth model inside OnlineHMR or Depth-Guided HMR for DVD? This is the real threat, because it reuses everything already built. Two things still fall short. DVD's depth on dynamic humans is unvalidated, since DVD reports no number on Bonn or Sintel on Lacuna. And neither pipeline feeds body shape back into depth scale. We run this swap as an explicit baseline. If it matches our full system, the contribution reduces to "use DVD depth," and we will report that.

Why not just smooth the joints or run physics optimization? Low-pass filters cause foot sliding during stance. Contact and Human Dynamics removes physically implausible forces but takes 30 to 60 minutes per 2-second clip and assumes flat ground. Our method targets root-depth drift, which neither approach addresses. We will report foot sliding to confirm we do not worsen it.

Proposed Experiments

The first check is diagnostic and decides whether the stitching half of the idea survives. On EMDB, which provides world-aligned video and depth, and on Bonn's dynamic indoor sequences, we fit DVD's global affine between overlapping windows. We then compare the residual inside SMPL-rendered human masks against the residual on the background, broken down by motion speed.

The second experiment evaluates depth on Bonn, Sintel and EMDB. We report AbsRel and δ<1.25 over the whole image and within human masks, plus a temporal metric: the mean absolute change in aligned depth on static background pixels across frames.

The third experiment evaluates pose on 3DPW, 3DPW-OC, EMDB-2 and Human3.6M. We report MPJPE, PA-MPJPE, acceleration error, WA-MPJPE, root translation error and foot sliding, with the cited baselines above.

The fourth experiment profiles latency. We report milliseconds per frame, peak GPU memory, and accuracy as a function of the number of iterations used by the gradient-fitting ablation, all on one GPU type.

The proposal is supported if four conditions hold together:

  • Human-region global-fit residuals clearly exceed background residuals.
  • Part-wise plus ruler stitching closes that gap.
  • World-frame drift falls below OnlineHMR and the DVD-swap baseline without hurting MPJPE.
  • The closed-form solve matches iterative fitting accuracy at under a tenth of its added cost.