Space-Time Plane Consensus for Mining Physically Grounded Assets from Interaction Videos

Simulators for robot learning are short on realistic objects because people still model them by hand. An obvious source of real objects is video of people or robots handling things on tables and floors. The difficulty is the place where the object touches the surface. Methods that reconstruct moving scenes either blur the object into the table at the contact or leave the object with a ragged, missing bottom. Methods that take objects from real footage have no notion of contact at all, so the exported objects float above or sink into the floor. This note proposes treating the support surface as something to estimate first and then reuse, not just discard. We fit the table or floor plane with RANSAC, the classic method that guesses a plane from three random points and keeps the guess most other points agree with. The change is that agreement is counted over the whole video rather than one frame. A point counts as "surface" only if it lies on the plane and stays put over time. Points inside the plane's thickness band that come and go are reclassified as foreground: the base of a sliding box, a cable lying flat, a hand pressing down.

The plane that results is then used three ways while the moving scene is reconstructed. It owns its own surface primitives, so no primitive can belong to both the object and the table. It acts as a barrier objects cannot pass through. And whenever an object rests on it, it closes off the object's unseen underside. The output is a set of watertight, metric-scale object assets, each with a known contact face and resting pose, ready to drop into a physics simulator. In two sentences: standard plane removal is static and over-deletes anything flat, so we make the plane fit time-aware and get both a clean surface and the objects' contact regions. We then use that plane as a hard geometric constraint during dynamic reconstruction, so the extracted objects come out physically grounded instead of floating or interpenetrating.

Core Claim

The claim is that one cheap geometric estimate fixes three failures that the literature currently treats separately, provided it respects time. The first is contact-zone tearing in 4D decomposition, meaning reconstruction of scenes that change over time. The second is floating and penetration in asset-driven reconstruction. The third is incomplete object undersides caused by the support occluding them.

The measurable consequences are lower object Chamfer error and near-zero penetration on human-object interaction data, and higher recall of low-profile foreground. The extracted assets should also settle in simulation in the same pose the real object took. We claim this beats a strong learned pipeline even when that pipeline is given the same plane after the fact.

Where This Sits in Prior Work

Plane fitting as preprocessing is ubiquitous. Examples include stripping tables for grasping (Yue Qiu et al., Borghetti Soares et al.), progressive sampling variants for cluttered supports (Chen Hu et al.), and sequential RANSAC for multi-plane structures (Pusztai et al.). The closest temporal use is Weakly and Self-Supervised Class-Agnostic Motion Prediction. It fits a ground plane in each LiDAR sweep and assumes the ground is static, so that motion learning needs no labels. That work motivates ours: it shows a geometric static/dynamic split is nearly free. It also exposes the limitation we target, since its fit is single-frame and single-plane, and anything moving within the inlier band is silently erased. Quatro++ likewise removes ground only to throw it away. We keep the plane and make it a participant in reconstruction.

On the reconstruction side, Split4D clusters Gaussian primitives (small 3D blobs used as rendering units) by shared trajectories to split moving objects from background. Its authors report that primitives get shared across hand-table and object-table contacts, producing tearing. This is the core evidence for the gap. CCGS shows that projecting Gaussians onto locally fitted planes removes floaters and sharpens boundaries, but only in static scenes. We borrow its projection mechanism and apply it to a time-consistent global support plane rather than local semantic patches.

ADSR is the strongest system connecting assets and dynamics. It generates meshes from clean canonical frames with a 3D generative model, tracks them, and optimizes only poses. Its authors state that it models no ground contact, so objects penetrate or float. It also cannot recover when the initial pose estimate is wrong. We build on its pose-only design and add the plane as a constraint and as a source of pose initialization (resting objects have two fewer free rotational degrees). GeoSim mines vehicle assets from driving logs but places them using HD maps, not estimated support surfaces. RoboGen motivates the whole effort: its retrieved assets fail on tight-tolerance tasks because their geometry and placement do not match real contact.

Research Program

The pipeline has three stages, and each is testable on its own.

Stage one: space-time plane consensus. All frames are lifted into a shared world frame using camera poses and depth, from RGB-D sensors or from predicted depth where only RGB exists. The space is voxelized. For every voxel $v$ we record its visibility count $n_v$ (frames in which it was in view and unoccluded) and its occupancy count $o_v$ (frames in which a point fell inside it). A candidate plane $\pi$, sampled from point triples drawn across frames, is scored by

$$S(\pi) = \sum_{v:\, d(v,\pi) < \epsilon} \left( \mathbb{1}\left[\tfrac{o_v}{n_v} \ge \tau\right] - \lambda\, \mathbb{1}\left[\tfrac{o_v}{n_v} < \tau\right] \right),$$

where $d(v,\pi)$ is the voxel's distance to the plane, $\epsilon$ is the inlier band, $\tau$ is the persistence threshold, and $\lambda$ penalizes transient points in the band. Sequential RANSAC repeats this to recover the floor, the tabletop, and shelves. The key output is not the plane alone. It is the set of in-band, non-persistent voxels, which are labeled "contact foreground" and returned to the object side. This directly addresses two limitations of plane subtraction: deletion of low-profile objects, and blindness to motion inside the band. Persistence is computed over visible frames only, so a region that is merely occluded by a hand is not mistaken for something that moved.

Stage two: plane-aware dynamic reconstruction. We reconstruct the scene with dynamic Gaussians. Objects are initialized from the foreground points and tracked with per-object rigid poses in the ADSR style. Three plane terms are added. First, each support plane is represented by its own flattened Gaussians, which are projected onto the plane after every densification step as in CCGS. Ownership is exclusive: an object Gaussian cannot be absorbed into the plane set, and the reverse is also forbidden. Second, a one-sided penalty $\sum_g \max(0, -d_s(g,\pi))^2$ is applied to object Gaussians $g$, where $d_s$ is the signed distance to the plane on its free side. This forbids passing through the plane. Third, contact intervals are detected: periods when an object's lowest points sit within $\epsilon$ of the plane and the object is still relative to it. During these intervals the object's lower boundary is closed by the plane, which supplies geometry for the underside that the camera never saw. Moving cast shadows on the plane are handled by giving plane Gaussians a per-frame brightness multiplier, kept low-dimensional so that the multiplier cannot absorb object geometry.

Stage three: asset export and physical check. Each object's Gaussians, together with its plane-closed base, are converted into a watertight mesh at metric scale, with a recorded contact face and resting pose. The assets are imported into the physics engine used by RoboGen. Two checks follow. One is whether each asset, dropped from slightly above its resting pose, settles into the pose observed in the video. The other is whether short real pushing trajectories are reproduced when the observed push is replayed.

Baselines

The naive reference point is the conventional pipeline: single-frame RANSAC subtraction with a fixed band, then the same dynamic reconstruction with no plane terms. It isolates what you get from "just remove the floor." We expect it to delete flat foreground and to leave floating assets. It is run on identical inputs with the same $\epsilon$ and the same number of optimization iterations.

The strongest existing method on dynamic human-object scenes is ADSR. On HOI-M3 it reports an object-wise Chamfer distance of 0.0493 cm, object-wise distance accuracy of 0.0446 cm, and whole-scene distance accuracy of 0.0843 cm, against 0.0734/0.0840/0.2165 cm for DreamScene4D (ADSR). We will run ADSR's released pipeline where available on the same sequences and report its numbers alongside the published ones. Beating it means lower object Chamfer and whole-scene accuracy than our own rerun of ADSR, together with a large reduction in penetration depth and floating gap. ADSR has no reported result for these two metrics, so we compute them for all methods.

For decomposition quality, the competitor is Split4D. It reports mIoU 0.893 on its multi-human data against 0.592 for SA4D. No reported result was found on Lacuna for its Neural3DV/SelfCap decomposition numbers. We will compare boundary IoU restricted to a band around contact regions, since that is where the claimed failure lives.

The decisive extra baseline is ADSR plus a post-hoc snap. It receives our space-time plane and, after tracking, translates each object along the plane normal per frame so it neither penetrates nor floats. It gets the same plane and the same compute. If it matches our method on all metrics, the in-loop constraint adds nothing beyond the plane estimate.

Ablations isolate the key ingredient. We swap space-time consensus for per-frame consensus with everything else held fixed. We separately remove the exclusive-ownership projection, the non-penetration term, and the plane closure. The persistence-based consensus is the claimed novelty, so the first ablation is the one that must show a clear gap on low-profile recall and contact-band IoU.

Why not just…?

Why not just segment objects with a 2D foundation segmenter and skip plane fitting? Per-frame masks are the obvious alternative. Split4D's motivation is precisely that video segmentation drifts and switches identities during fast hand-object motion, and that mask errors get baked into 3D. Masks also say nothing about where the support is, so they cannot supply non-penetration or underside closure. We will still run a mask-driven variant: tracked 2D masks lifted to 3D, then the same reconstruction. This tests whether learned masks recover the contact band as well as temporal persistence does. If they do, the consensus stage could be replaced, but the plane would still be needed for stages two and three.

Why not just snap ADSR's output onto the floor afterward? This is the real threat, and it is why the snap baseline exists. Snapping fixes the vertical offset at resting frames. It cannot repair an underside that was never observed or a tilted resting orientation. It also cannot fix poses that got stuck in a bad optimum because the optimizer was free to pass through the table. Our prediction is that snapping closes most of the penetration gap but not the Chamfer gap or the simulation resting-pose gap. If snapping matches us on Chamfer as well, we will report that the contribution reduces to the consensus estimator.

Why not just generate assets with a text- or image-to-3D model? RoboGen documents failures on precision tasks with retrieved assets. ADSR's generated meshes have no scene-level contact or scale. Our simulation resting-pose and push-replay tests compare mined assets against ADSR's generated meshes for the same objects. This gives a direct measure of whether real geometry matters downstream.

Why not just use robust losses or delayed densification? RobustNeRF reaches 30.87 dB against 25.22 dB for Mip-NeRF 360 on distractor scenes by discarding high-residual pixels. RobustSplat delays densification. Both recover the background by throwing the moving object away, which is exactly what we want to keep, and both acknowledge trouble with moving shadows. They are therefore baselines only for background quality. Here we will check that our plane Gaussians are at least as clean in held-out views.

Proposed Experiments

The main benchmark is HOI-M3 (HOI-M3). There we report ADSR's metrics plus penetration depth, floating gap, and contact-band boundary IoU. Split4D's evaluation sequences provide the decomposition comparison.

No existing benchmark stresses low-profile foreground, so we will record a tabletop RGB-D set with ground-truth scanned objects. It will include flat tools, cables, and thin boxes being slid, lifted, and set down. On this set we measure recall of foreground points within $\epsilon$ of the table. This is the cleanest test of the space-time consensus, because single-frame RANSAC should score near zero there by construction.

Downstream, all extracted assets go into the RoboGen simulator for the resting-pose test (rotation and translation error versus the observed rest) and for push replay. Push replay follows DSR-Net's real-robot pushing protocol (DSR-Net), which reports 3.33 cm motion-prediction error for its learned model; we use that only as a reference scale.

The decisive checks, in order of importance, are three. First, space-time consensus must beat per-frame consensus on low-profile recall and contact-band IoU. Second, the full method must beat ADSR-plus-snap on object Chamfer and simulated resting pose, not just on penetration. Third, mined assets must reproduce real resting poses and push outcomes better than generated meshes of the same objects. Failure on the second check would sharply narrow the claim, and we would report it as such.