A 3D convolutional network treats a video clip as a block of numbers indexed by time, height and width, and slides small cubic filters through that block. This design quietly assumes that pixel $(x, y)$ in frame $t$ and pixel $(x, y)$ in frame $t+1$ show roughly the same piece of the world. That holds for a camera on a tripod. It fails for a head-mounted or robot-mounted camera, where every head turn shifts the whole image.
Under such motion the filters' "time" direction mixes unrelated surfaces. The skip connections that U-Net-style decoders use to carry fine detail around the compressed middle of the network also carry misaligned detail. The result is blurred boundaries exactly where they matter most: thin handles, fingers and contact edges. We propose a simple correction. Before every temporal convolution and every skip connection, features from neighbouring frames are warped into the current frame's viewpoint using estimated depth and camera motion. The temporal axis then means "the same 3D point over time", and the only motion left inside the volume is the motion of objects themselves.
We test this on egocentric hand-object video. Rendered views of professionally built articulated assets serve as a controlled source of training and evaluation data. The practical goal is a dense perception front end that can segment moving parts and their motion from casual handheld video. That is a first step toward producing simulation assets without hand modelling.
Core Claim
The fixed-grid temporal axis of 3D convolution is a hidden static-camera assumption. Removing it at the level of features, at every scale, should do two things. It should recover the motion-sensitive features that 3D convolutions are good at, and it should turn skip connections from a source of misaligned detail into a way of pooling detail across several frames. A thin structure blurred in one frame can then be sharpened by its aligned neighbours.
Concretely, let $F_{t'}$ be the feature map of frame $t'$ at some network scale, $D_t$ the depth of frame $t$, $K$ the camera intrinsics, $T_{t\to t'}$ the relative camera pose, and $\pi$ perspective projection. For each pixel $p$ of frame $t$ (with $\tilde p$ its homogeneous coordinates), we form the aligned neighbour feature
$$\tilde F_{t'\to t}(p) = F_{t'}\big(\pi(K\, T_{t\to t'}\, D_t(p)\, K^{-1}\tilde p)\big),$$
sampled bilinearly. We run the ordinary $3\times3\times3$ convolution on the stack $\{\tilde F_{t'\to t}\}$ over a short window of $t'$. The same aligned stacks feed the skip connections into the decoder.
The prediction is falsifiable. As camera speed increases, the standard architecture's dense accuracy and boundary accuracy should fall, while the aligned version should stay roughly flat. The gain should concentrate on thin structures.
Where This Sits
The method we reuse. Deep End2End Voxel2Voxel Prediction (V2V) is a 3D-convolutional encoder with 3D transposed-convolution decoders and long skip connections for per-voxel prediction. It reports 76.0% accuracy on GATECH segmentation against 55.7% for a 2D baseline. We keep its architecture and change only how features are indexed in time.
What motivates the change. V2V also documents two constraints we must respect. Memory limits it to 16-frame windows, and pre-training on action recognition hurt its optical-flow performance. We therefore train from scratch or from dense-task initialisations rather than from action classifiers. Learning Spatiotemporal Features with 3D Convolutional Networks established the homogeneous 3D kernel that both V2V and our model use. Neither paper treats a moving camera.
What supplies the geometry. SpatialTrackerV2 estimates scale-aligned depth and camera pose in a feed-forward pass and uses them for sparse 3D point tracking. We use its front end, or ground truth where available, as the source of $D_t$ and $T_{t\to t'}$. Our output is dense volumetric prediction rather than tracks.
Closest existing work. The nearest system is EgoHOI, a 14B-parameter video diffusion world model for egocentric hand-object rollouts. It uses a temporal 3D-convolution stack over hand mesh renders and encodes camera motion as Plücker ray fields: per-pixel ray directions supplied as extra input. EgoHOI tells the network how the camera moved and hopes it learns to compensate. It is a generator, and it depends on pre-estimated hand poses. Our approach differs in three ways. We change where the convolution samples instead of adding an input channel. We address discriminative dense prediction. We test cheaply whether explicit alignment beats ray conditioning.
The detail-preservation lesson. gen2seg shows that keeping a full high-resolution decoder preserves thin structures that invariance-trained models lose: 51.4 versus 16.8 mIoU against SAM on iShape. That work is single-image. We carry the lesson into video, where the extra resource is aligned neighbouring frames.
Why assets. ArtVIP shows the cost of the current route to high-fidelity assets. Its 992 articulated digital twins required professional artists. R2RGen generates data from real demonstrations but is explicitly limited to fixed-base cameras and rigid objects. We use ArtVIP's assets not as the end product but as a source of exact ground truth for moving parts filmed by a moving camera.
Research Program
The model. We build one V2V-style network with three switchable modes:
- no alignment (the original);
- alignment at the input only;
- alignment at every encoder scale and every skip connection.
Depth and pose come either from ground truth, as an oracle, or from SpatialTrackerV2's front end, as the realistic setting. Pixels that leave the field of view or become occluded are detected with a depth-consistency test and masked. The kernel never averages over garbage.
The output heads. The network predicts three things per voxel: a hand, object or moving-part label; a residual 3D motion vector (the scene motion left after camera motion is removed); and a boundary map.
The data. We train on two sources. The first is HOT3D-Clips (1,364 training and 152 evaluation clips), described in HOT3D. Because HOT3D provides 90 Hz motion-capture poses and 33 high-resolution object meshes, dense masks and per-pixel motion can be rendered exactly for real egocentric video. The second is synthetic handheld-camera renders of ArtVIP articulated assets: drawers, doors and lids being actuated. Camera speed is controlled, and ground truth for part masks and joint axes is exact.
The asset-facing analysis. A small downstream module turns aligned part masks and residual motion into a rigid part segmentation and a fitted joint axis per articulated object. This tests whether better dense perception translates into better automatically recovered articulation. We do not claim a full asset generator. We test the first measurable link in that chain.
Baselines
Naive reference: unaligned V2V. The reference is V2V exactly as published in Deep End2End Voxel2Voxel Prediction, trained on our data with our heads. Everything else in the comparison is held fixed: parameter count, 16-frame window, input resolution, training data, and hyperparameter search budget. It shows what the static-camera assumption costs. On egocentric dense segmentation there is no reported result on Lacuna for V2V or any 3D-CNN on HOT3D. Its only reported number, 76.0% on GATECH, comes from a different, largely static-camera domain and is not comparable. We will report our own reproduction.
Ray-conditioned V2V. This is the same network given Plücker ray fields as extra input channels, following EgoHOI's Ego-Motion Embeddings (EgoHOI). It is the strongest competitor on the central question because it uses the same pose information without changing the convolution. If it matches the aligned model, the alignment is unnecessary. EgoHOI reports a hand Missing Ratio of 5.84%, against 14–20% for its baselines, on HOT3D-Clips. That metric is for generated video, so it frames the domain rather than serving as a target. The aligned model beats this baseline if it shows a statistically significant gain in boundary F-score and thin-part IoU on HOT3D evaluation clips across three seeds.
Strong per-frame 2D pipeline. The third baseline is a gen2seg-style single-frame segmenter (gen2seg) with a full generative decoder, fine-tuned on the same frames. Motion comes from SpatialTrackerV2 point tracks densified by interpolation. This represents "skip the 3D convolution entirely".
Key-ingredient ablations. Two ablations isolate what matters. One compares alignment at input only against alignment at all scales and skips. The other compares oracle pose against estimated pose, separating the value of the mechanism from errors in the pose estimator.
Why not just…?
Why not just stabilise the video first and run ordinary V2V? This is the most serious threat. Classical stabilisation warps each frame with a single homography, a planar image warp. That is exact for pure camera rotation but wrong under translation, where near objects shift more than far ones. In egocentric manipulation the hands and objects are tens of centimetres from the camera, so translation-induced parallax is largest exactly where the details are. We include homography-stabilised V2V as an explicit baseline and sweep camera translation separately from rotation in the ArtVIP renders. If homography stabilisation matches depth-based alignment at realistic translation speeds, we will report that the cheap fix suffices and drop the per-scale machinery.
Why not just give the network the camera rays and let it learn compensation? EgoHOI does this, and it clearly helps generation. But a $3\times3\times3$ kernel can only compensate for displacements within its receptive field. Head rotations can move content far beyond a few pixels per frame at the finest scales, which is precisely where skip connections carry detail. The ray-conditioned baseline, binned by camera speed, tests this directly. The expected signature is a deficit that grows with speed and is largest on boundary metrics. If no such deficit appears, the claim fails.
Why not just use per-frame 2D segmentation plus a point tracker? Single-frame models cannot borrow evidence from other frames when a thin part is blurred or partly occluded by the hand. That is the situation in which DECO (DECO) needed tactile sensors, gaining 20 absolute points on occluded contact tasks. A per-frame pipeline also has no dense motion of its own. TAPVid-3D (TAPVid-3D) shows lifted 3D tracking scoring only in the low 20s of 3D Average Jaccard, against the 50s–60s in 2D. The per-frame baseline measures whether multi-frame pooling adds anything beyond a strong single-image decoder. If it ties on thin-part IoU, the detail-pooling half of our claim fails.
Why not just use temporal attention instead of convolution? Attention can in principle learn correspondences without being told the geometry, as SpatialTrackerV2's own architecture does for tracks. But its cost grows quadratically with the number of space-time tokens at full resolution. We add a compute-matched space-time attention baseline at the coarsest scales only, which is the affordable version. We report accuracy at equal GPU memory.
Proposed Experiments
Camera-speed sweep. On ArtVIP renders, camera rotation and translation speed are varied independently across five levels, with object articulation held fixed. For every method we plot part IoU, boundary F-score at a two-pixel tolerance, and residual-motion endpoint error against speed. The decisive check is the slope. Unaligned and ray-conditioned V2V should degrade with speed while the all-scale aligned model stays flat. The gap between oracle and estimated pose bounds how much depends on SpatialTrackerV2's accuracy.
Real egocentric transfer. On HOT3D evaluation clips we report:
- hand and object mask IoU;
- boundary F-score;
- IoU on a thin-structure subset (pixels within a few pixels of mesh silhouette edges or on handle-like parts);
- per-pixel 3D motion error against motion-capture ground truth.
We compare models trained on HOT3D alone against models trained on HOT3D plus ArtVIP renders. This tests whether manually built assets transfer as training signal to real moving-camera video. For context on the regime, HOT3D reports monocular 3D lifting recall of 14.3% at 5 cm, against 64.4% with stereo matching. This motivates using metric depth from a learned front end rather than naive monocular depth.
Point-tracking sanity check. The residual-motion head is integrated into trajectories on TAPVid-3D and scored with 3D Average Jaccard and APD3D. SpatialTrackerV2 is reported only as "up to 30%" better than prior methods, with no absolute value in the file. We will therefore run it ourselves rather than quote a target. We do not expect to beat a dedicated tracker. The aim is to confirm that the aligned dense motion is geometrically sound.
Articulation recovery. On held-out ArtVIP assets, never seen during training, we fit a joint axis from the predicted part masks and residual motion. We measure axis angle error and pivot-point error against ArtVIP's ground truth, as a function of camera speed. This is the experiment that connects perception to the asset bottleneck. If the aligned model's improvements in IoU do not reduce joint-axis error, the asset-facing motivation is weakened. We would report the method as a dense-perception contribution only.
The whole program fits in 16-frame windows on single-GPU memory budgets, since warping adds sampling cost but no parameters. It can therefore be run by a small group without diffusion-scale compute.
References
- Deep End2End Voxel2Voxel Prediction
- Learning Spatiotemporal Features with 3D Convolutional Networks
- SpatialTrackerV2: Advancing 3D Point Tracking with Explicit Camera Motion
- Egocentric World Model for Photorealistic Hand-Object Interaction Synthesis
- gen2seg: Generative Models Enable Generalizable Instance Segmentation
- ArtVIP: Articulated Digital Assets of Visual Realism, Modular Interaction, and Physical Fidelity for Robot Learning
- R2RGEN: Real-to-Real 3D Data Generation for Spatially Generalized Manipulation
- HOT3D: Hand and Object Tracking in 3D from Egocentric Multi-View Videos
- DECO: Decoupled Multimodal Diffusion Transformer for Bimanual Dexterous Manipulation with a Plugin Tactile Adapter
- TAPVid-3D: A Benchmark for Tracking Any Point in 3D