Rigid Convex Groups for Fast Dynamic Scene Rendering

Methods that rebuild a 3D scene from photos so it can be viewed from new angles mostly assume nothing in the scene moves. The dominant fast method, 3D Gaussian Splatting (3DGS), represents a scene as millions of small fuzzy ellipsoids that are projected and blended onto the screen. When things move, current systems give each ellipsoid its own trajectory. The cost of describing motion then grows with the number of ellipsoids, and flat or sharp-edged surfaces need a great many of them. This proposal starts from a plain observation. Much of what moves in everyday and driving footage is rigid: cars, doors, chairs, boxes. A rigid object built from a few sharp-edged convex shapes needs one motion path, not thousands.

We propose to represent a dynamic scene with smooth convex primitives (each one the rounded convex hull of a handful of 3D points). These are grouped into rigid bodies, and each body follows a single smooth trajectory over time. Which convexes belong to which body is learned jointly from two signals: whether they move together, and visual features lifted from a 2D image foundation model. A small set of ordinary deformable Gaussians is kept only for genuinely non-rigid content such as people and cloth. The bet is that this layout gives dynamic reconstructions three things at once: far fewer motion parameters, crisper boundaries where moving objects touch the background, and frame rates and memory use that stay flat as videos get longer.

Core Claim

The claim is that in scenes dominated by rigid motion, primitive shape and motion structure should be chosen together. Convex primitives make rigid grouping pay off, because one convex covers a flat face that would otherwise take hundreds of Gaussians. Rigid grouping in turn makes convexes usable in dynamic scenes, because their rendering cost is mostly per-primitive and per-group rather than per-frame-per-point. Neither half alone gives the full gain. Gaussians with rigid grouping still pay the packing cost on flat surfaces. Convexes with independent per-primitive motion pay for many redundant trajectories and drift apart at object edges.

Concretely, convex $k$ has fixed canonical vertices $p_{k,1..m}$ and a soft assignment $a_{k,g}$ to rigid groups $g$, where $m$ is the number of vertices per convex. Its vertices at time $t$ are

$$x_{k,i}(t) = T_{g^*(k)}(t)\, p_{k,i},$$

where $T_g(t)$ is a rigid rotation-plus-translation for group $g$, parameterized by a cubic spline in time, and $g^*(k)$ is the group with highest assignment (soft-blended during training, hardened at the end). Rendering a frame therefore needs one spline evaluation per group and one small matrix multiply per vertex, followed by the existing convex rasterizer.

Grounding in Prior Work

Two papers supply the representation. 3D Convex Splatting replaces Gaussians with smooth convexes, projected with a convex-hull scan and filled with a soft indicator. It reports under 15% of 3DGS memory at 30 to 80+ FPS, with higher fidelity. 3D Superquadric Splatting does the same with superquadrics but lacks a closed-form projection or a real-time rasterizer. Both are static only, which is the gap we target. We build on convexes rather than superquadrics because convexes already have a fast CUDA path, and a rigid transform of hull vertices preserves convexity exactly.

Three papers define the motion side and set the bar. From Tokens to Nodes drives Gaussians with sparse spline control nodes placed using DINOv2 features. It reports 39 minutes of training on HyperNeRF and a 25MB model. Split4D gives each Gaussian a velocity and an identity feature, learned contrastively against per-frame 2D masks, and reports blurring at contact zones. Hierarchy UGP organizes 4D Gaussians into a hierarchy for kilometer-scale driving scenes, relying on external trackers whose failures hurt quality.

All three keep ellipsoidal primitives. Their motion compression is achieved by sharing control signals, not by making each primitive cover more surface. We borrow the semantic-guidance idea from Tokens to Nodes and the contrastive identity loss from Split4D. Our change is to attach both to convex primitives, which are far fewer in number.

Three papers constrain the design. PCF-Lift shows that 2D masks are view-inconsistent and models lifted features with uncertainty. That method is static and heavy, so we use deterministic per-convex features. We treat group assignment as soft, and lean on motion consistency where semantics are unreliable. RobustSplat shows that densification turns moving objects into floating debris; this is why our densification step first tries to explain a high-error region with a rigid moving group before spawning new primitives. NeRF On-the-go shows that semantic uncertainty methods mistake reflections for motion. A rigid-motion test gives an independent check: a reflection does not move as a rigid body with the geometry around it.

Research Program

The work has three parts.

Rasterizer. We extend the convex CUDA rasterizer with a per-frame vertex transform stage that applies group transforms before projection. We profile how cost splits between this transform, the hull scan, and blending as group count, convex count and sequence length vary.

Grouping. Each convex carries a low-dimensional feature. It is supervised by DINOv2 features projected from training views and by a Split4D-style contrastive loss on 2D masks. A motion-consistency term pulls convexes together when their best-fit rigid motions agree. Groups are initialized by clustering these features, then merged and split during training: groups whose trajectories coincide are merged, and groups whose rigid fit leaves a large residual are split.

Non-rigid fallback. Regions where the rigid residual stays high after splitting are handed to a node-driven deformable Gaussian layer in the style of Tokens to Nodes. The fraction of the scene that ends up in this layer is itself a reported result. It tells us how rigid a benchmark really is.

Baselines

The naive reference point is static 3D Convex Splatting trained directly on dynamic video, with no temporal parameters. It shows how badly the static assumption fails, and it sets the floor for memory and frame rate.

The strongest existing dynamic method on monocular, object-scale video is From Tokens to Nodes. It is the strongest competitor because it already compresses motion through sparse semantic control and is fast to train. The file records its 39-minute HyperNeRF training time and 25MB size. We found no reported HyperNeRF image-quality number on Lacuna for it.

On multi-object decomposition, the strongest competitor is Split4D, the closest work on tracker-free identity binding. On large driving scenes, it is Hierarchy UGP. No reported PSNR results were found on Lacuna for either.

All baselines are run from their released settings on the same frames, resolution and camera poses. Each gets the same GPU type and the same hyperparameter-search budget in GPU-hours, with each method's search starting from its published defaults. Beating a baseline means matching its novel-view image quality while using markedly less memory and higher rendering FPS on long sequences. We also report boundary accuracy at contact zones, measured where object masks are available, and whether a claimed win holds across all scenes or only on rigid-heavy ones.

Four ablations isolate the key ingredients:

  1. Independent per-convex trajectories, with grouping removed.
  2. Grouping driven by motion alone, with semantic features removed.
  3. The same rigid grouping applied to Gaussians instead of convexes. This ablation directly tests whether primitive shape matters.
  4. A hybrid of static 3D Convex Splatting for the background plus Tokens-to-Nodes Gaussians for everything that moves. This is the main threat, discussed next.

Why not just…?

Why not just use static convexes for the background and existing dynamic Gaussians for everything that moves? This is the most serious alternative, and it may capture most of the gain on scenes where moving content is mostly people. The proposal's added value is confined to rigidly moving objects: vehicles, opened doors, moved furniture. On those objects the hybrid pays the Gaussian packing cost on flat moving faces and blurs their edges against the background. We run this hybrid as a full baseline and stratify results by the fraction of rigid moving content. If the gap is negligible on driving scenes, where rigid motion dominates, the claim fails and we will say so.

Why not just compress Gaussian motion more aggressively, as in From Tokens to Nodes? Sparse control nodes reduce motion parameters, but every Gaussian is still stored and rasterized, and flat surfaces still need many of them. The rigid-grouped-Gaussians ablation measures how much of our gain comes from grouping alone. If it matches the full method, convexes are unnecessary for dynamics.

Why not just segment moving objects with a 2D tracker and fit a rigid transform per tracked object? That is roughly what Hierarchy UGP does, and the file notes its quality drops when trackers fail on unexpected motion. Our grouping uses tracking-free motion consistency, so we test robustness by comparing against a tracker-based variant on sequences with occlusion and abrupt motion.

Proposed Experiments

Static fidelity is checked first on Mip-NeRF 360, where 3D Convex Splatting was evaluated, so that adding motion machinery does not degrade static quality. We found no reported PSNR result on Lacuna for 3D Convex Splatting on Mip-NeRF 360, so we will rerun it under our protocol. Dynamic evaluation uses HyperNeRF for object-scale monocular motion and the large urban driving setting of Hierarchy UGP for rigid-heavy scenes. Distractor-filled casual captures of the kind RobustNeRF and NeRF On-the-go target test whether explaining motion beats discarding it.

We report PSNR, SSIM and LPIPS on held-out views. We also report rendering FPS and GPU memory as a function of sequence length, training time, primitive and group counts, and the fraction of content routed to the non-rigid layer.

Two results decide the outcome. The first is whether memory and FPS stay flat as sequences lengthen while quality matches From Tokens to Nodes and Split4D. The second is whether the full method beats both the hybrid baseline and the rigid-Gaussian ablation on rigid-heavy driving scenes. If either check fails, the idea reduces to a static result plus a known dynamic one.