When someone opens a drawer or swings a laptop lid in front of a camera, the motion itself shows which parts are separate and how they are hinged. Current methods that recover articulated 3D objects from such video exploit this "common fate" signal: points on one rigid part move together. They do so by fitting a fresh 3D model to every video, which takes tens of minutes per clip. Feed-forward 3D reconstruction is the opposite approach. It trains one network on many scenes so that at test time it outputs a 3D model in a single pass, in milliseconds to seconds. It has so far produced either static scenes or unstructured animations, never parts and joints from raw video. We propose a network that reads a short interaction video and directly outputs 3D Gaussians (small colored ellipsoids that can be rendered from any viewpoint), grouped into rigid parts, each with a joint type, axis, and per-frame joint angle or offset.
The central bet is that these outputs should form a motion bottleneck: every frame must be explained by moving whole parts along a few joint parameters. That structure does two jobs. On synthetic renders it gives exact supervision. On real, unlabeled interaction videos it turns photometric and flow re-rendering losses into a self-supervised signal that forces grouping by common fate. This lets the model adapt from synthetic to real without labels. Motion evidence from optical flow enters the 3D Gaussians through cross-attention, in a separate stream from appearance. The point of the separation is to keep the network from segmenting parts by texture, a shortcut that holds in synthetic assets and fails on real, uniformly painted furniture.
Core Claim
A feed-forward model trained only on synthetic articulated renders, with a rigid-part-and-joint output head and a separate motion stream fused into 3D, can recover part segmentation and joint axes from real interaction video in about a second. Self-supervised re-rendering through the kinematic bottleneck on unlabeled real videos should close much of the remaining gap. The claim is falsifiable at three points. A pipeline built from existing pieces could match it. A version without the separate motion stream could transfer equally well. A version with free per-frame Gaussians instead of rigid parts could adapt equally well to real data. Each is tested below.
Where This Sits
Four feed-forward systems define the starting point. pixelSplat predicts 3D Gaussians from image pairs using epipolar cross-attention and per-pixel depth distributions. We reuse its encoder design for the canonical geometry. L4GM trains on 12 million synthetic multi-view clips (Objaverse-4D) and outputs animated Gaussians from monocular video in about 3 seconds. It transfers zero-shot to real video, reaching 0.12 LPIPS and 691.87 FVD on Consistent4D. Its output, however, has no parts, no joints, and cannot be re-posed. Particulate is the closest kinematic precedent. It predicts parts, a kinematic tree, and joint axes in about 10 seconds using learnable part queries over a point cloud, and reaches 0.286 gIoU on Lightwheel against 0.144 for Articulate AnyMesh. It needs a clean static mesh and never sees motion, so a door hinged on the left and one hinged on the right can look identical to it. We borrow its part-query and axis-voting design. Our input differs: video rather than a finished mesh, with motion as primary evidence.
The motion-based methods motivate the problem and supply the self-supervision. FreeGaussian links RAFT optical flow to Gaussian displacement analytically and clusters moving Gaussians, at about 30 minutes per video. MovingParts groups particles by shared rigid transforms in a dynamic NeRF, at about 26 minutes per scene. PD²GS fits Kabsch rigid transforms to Gaussian trajectories across captured states. Articulation in Motion adds a static-during-motion detector so that a newly revealed drawer interior is not mistaken for a moving part. REACTO uses quasi-rigid skinning for casual video. All five are per-scene optimizers. We take their rigid-grouping logic and move it from the test-time objective into the network's output structure and training loss.
The bottleneck idea has an older precedent in Visual Dynamics. There, predicting future frames through a low-dimensional motion code made the code split the scene into independently moving components.
Two papers constrain the design. Tangemann et al. show that deep optical flow models lean on appearance: they collapse below 0.15 IoU on random-dot stimuli, while appearance-free motion-energy filters reach 0.600. On clean natural video, however, deep flow wins (about 90% versus 75.9%). We therefore use deep flow but do not let the model depend on it alone. The appearance stream must be able to take over where flow fails, on specular or textureless surfaces. Rank-targeted fusion documents that multi-modal networks collapse onto their dominant modality. We monitor this directly in our two-stream design. The domain-randomization result of Khirodkar et al. supplies the synthetic-data recipe: randomized, non-photorealistic textures push models onto geometry.
Research Program
The model. It takes $T$ frames (8 to 16) of an object being articulated, with camera poses. Poses come from the renderer on synthetic data and from SLAM or the capture rig on real data. An appearance encoder in the style of pixelSplat predicts canonical Gaussians from the first frame or two. A motion encoder turns flow between consecutive frames, plus frame-difference features, into a separate set of 2D tokens.
Fusion follows the pattern of SGFormer, in which 3D points query 2D tokens. Here each Gaussian projects into every frame and cross-attends to the motion tokens near its projection, gathering evidence about how it moved. $K$ part queries, as in Particulate, then attend to the fused Gaussians. They predict a soft assignment of each Gaussian to a part. One slot is reserved for the static body, and newly exposed interiors are trained to fall into it, following the AiM insight. Each part query also outputs a joint type (revolute or prismatic), an axis, and a per-frame joint state.
The rendering loss. Frame $t$ is rendered by moving each Gaussian $g$ with the rigid transform of its assigned part:
$$\mathcal{L}_{\text{self}} = \sum_{t=1}^{T} \Big\| I_t - R\big(\{\, A_k(\theta_{k,t})\, g \,\}_{g \in \text{part } k}\big) \Big\|_1 + \lambda \sum_{t} \big\| F_t - \hat F_t \big\|_1 .$$
Here $I_t$ is the observed frame, $R$ is the Gaussian rasterizer, and $A_k(\theta_{k,t})$ is the rigid transform of part $k$ given its joint state $\theta_{k,t}$ and predicted axis. $F_t$ is the observed optical flow. $\hat F_t$ is the flow induced by the predicted part motions, and $\lambda$ weights the flow term. Because motion can only be expressed through a handful of joint parameters, the cheapest way to lower this loss is to group Gaussians that move together.
Training. It runs in two phases. The first is supervised: interaction videos rendered from PartNet-Mobility training categories under domain randomization, with losses on parts, joint type, axis, and state, plus $\mathcal{L}_{\text{self}}$. One randomization is specific to this problem. Textures are sampled so that they cross part boundaries (a single pattern spanning door and frame), with part-distinct textures only some of the time. This breaks the correlation between material edges and kinematic edges. The second phase adds unlabeled real interaction videos with only $\mathcal{L}_{\text{self}}$ applied, mixed with synthetic batches so the supervised structure is not forgotten. No dataset of such videos with an identifier is available, so we would combine the InterReal videos, any released Aria sequences from AiM, and a few hundred smartphone clips of household objects collected by us, similar to REACTO's casual captures. Evaluation objects are held out from this pool.
Optional refinement. A few hundred gradient steps of $\mathcal{L}_{\text{self}}$ at test time let us report a speed-accuracy curve that runs from pure feed-forward to optimization-based methods.
Baselines
The naive reference point is a static-geometry predictor with no motion. We use Particulate given the ground-truth mesh, which is generous to it, on held-out PartNet-Mobility and Lightwheel assets. On real objects we give it a mesh reconstructed from the video's first frames. Its reported Lightwheel figure is 0.286 gIoU (Particulate). We will reproduce it under our split. Beating it means higher gIoU on Lightwheel using only rendered interaction video, with no mesh. The largest margin is expected on a curated subset whose joint location is ambiguous from static shape alone.
The strongest feed-forward competitor is a composed pipeline. L4GM produces animated Gaussians. We then apply AiM-style sequential RANSAC with Kabsch fitting to Gaussian trajectories, or to flow-lifted point tracks if L4GM's Gaussians do not keep identity across frames. Both stages use released code with no retraining, and its trajectory-clustering thresholds get the same validation tuning budget as our hyperparameters. It is the most serious threat because it already transfers zero-shot to real video and adds only a few seconds of clustering. Beating it means lower axis-angle error and higher part IoU on RS-Art and held-out PartNet-Mobility, at comparable runtime.
The strongest accuracy reference is per-scene optimization. We run Articulation in Motion and FreeGaussian on the same videos. Reported figures come from different input protocols and are not directly comparable. AiM reports 0.27° oven axis-angle error, down from a previous best of 5.39°, using a static scan plus video. PD²GS reports 0.21° on PartNet-Mobility Storage from multi-state multi-view captures. FreeGaussian reports 29.133 PSNR on InterReal. No reported result was found on Lattice/Lacuna for any of these methods on RS-Art with numbers. We do not expect pure feed-forward inference to beat them. Our criteria are two. First, median axis error within a few degrees of theirs at roughly a thousandth of the runtime. Second, feed-forward plus short refinement matching or beating them at a tenth of their runtime.
Three ablations isolate the key ingredients. The first removes the motion stream and feeds raw frames to a single video transformer, testing whether separate motion fusion matters. The second replaces the rigid-part output with free per-frame Gaussian offsets while keeping the same real-video losses, testing whether the kinematic bottleneck is what makes self-supervision useful. The third trains on synthetic data only, with no real-video phase, isolating the contribution of self-supervised adaptation.
Why not just…?
Why not just run L4GM and cluster its trajectories? This is the alternative most likely to work. If it matches us, the right conclusion is that kinematic structure should be extracted post hoc rather than predicted. Two concerns are specific. L4GM is trained on floating objects with pre-rigged animations (L4GM), so it has never had to keep a revealed drawer interior static. AiM shows that this "new geometry" case breaks trajectory clustering even inside per-scene optimizers. Clustering is also downstream of reconstruction errors it cannot correct. The composed-pipeline baseline settles this directly. We will report results split by whether the motion reveals hidden geometry, since that is where we expect any advantage to appear.
Why not just reconstruct a mesh and run Particulate? Static shape underdetermines many joints: hinge side, push versus pull, a lid versus a sliding cover. Particulate reasons only from shape (Particulate). It also inherits every artifact of the reconstruction step, including the floaters and missing surfaces that feed-forward reconstructors produce. On categories where shape fully determines the joint, it may tie or win. The ambiguous-joint subset is designed to show where motion is necessary. We will report both subsets rather than an average.
Why not just run AiM or FreeGaussian and accept the wait? For offline digitization, this is a fine answer. Interactive use, robot exploration, and large-scale asset capture cannot spend 25 to 30 minutes per clip. The question is therefore the speed-accuracy frontier, not accuracy alone. The refinement curve shows whether a feed-forward start lets optimization converge in seconds. If it does not, the feed-forward model is only a fast approximation, and we will say so.
Why not just domain-randomize harder and skip real-video self-supervision? Domain randomization alone carried car detection across the gap (Khirodkar et al.). Real interaction footage, however, adds specularity, fisheye distortion, and hands in the frame (Articulation in Motion), which procedural renderers rarely reproduce. The synthetic-only ablation measures exactly how much real-video adaptation adds. If the gain on RS-Art is negligible, the second training phase should be dropped.
Proposed Experiments
The synthetic benchmark holds out entire PartNet-Mobility categories. We render interaction videos under both part-distinct and boundary-crossing textures and report part mIoU, joint-type accuracy, axis-angle error, and axis-position error.
The texture-shortcut test is the most diagnostic single experiment. Each model trained on standard textures is evaluated on boundary-crossing textures. If the RGB-only ablation drops sharply and the motion-stream model does not, that confirms the network was otherwise segmenting by texture.
Real-world transfer uses RS-Art, which has 18 objects with URDF ground truth, by rendering or using captured interaction sequences, and reports the same kinematic metrics. InterReal reports re-posed novel-view PSNR against FreeGaussian's 29.133. Lightwheel assets are rendered into videos for gIoU against Particulate's 0.286.
Two further checks probe failure modes. A flow-corruption sweep injects blur and low texture to test whether the appearance stream compensates when flow fails, as Tangemann et al. predict it must. We also track the attention mass each stream receives across training, to detect the modality collapse described in the rank-targeted fusion work.
The proposal succeeds if three things hold together on RS-Art: the full model beats the L4GM-plus-clustering pipeline on axis error; removing the rigid-part bottleneck erases the gain from real-video adaptation; and removing the motion stream fails the texture-shortcut test. Failing any one of these narrows the claim accordingly.
References
- PixelSplat: 3D Gaussian Splats from Image Pairs for Scalable Generalizable 3D Reconstruction
- L4GM: Large 4D Gaussian Reconstruction Model
- Particulate: Feed-Forward 3D Object Articulation
- FreeGaussian: Annotation-free Control of Articulated Objects via 3D Gaussian Splats with Flow Derivatives
- MovingParts: Motion-based 3D Part Discovery in Dynamic Radiance Field
- PD$^{2}$GS: Part-Level Decoupling and Continuous Deformation of Articulated Objects via Gaussian Splatting
- Articulation in Motion: Prior-free Part Mobility Analysis for Articulated Objects By Dynamic-Static Disentanglement
- REACTO: Reconstructing Articulated Objects from a Single Video
- Visual Dynamics: Probabilistic Future Frame Synthesis via Cross Convolutional Networks
- Object segmentation from common fate: Motion energy processing enables human-like zero-shot generalization to random dot stimuli
- Countering Multi-modal Representation Collapse through Rank-targeted Fusion
- Domain Randomization for Scene-Specific Car Detection and Pose Estimation
- SGFormer: Semantic-Geometry Fusion Transformer for Multi-modal 3D Panoptic Segmentation