Driving perception systems are increasingly trained by copying knowledge from 2D image models into 3D. The image models are vision foundation models: networks such as CLIP, DINOv2 and SAM, pretrained on billions of images, that produce rich per-pixel features. The copying works best when many cameras or frames see the same 3D point, because the 3D model must then agree with all views at once and the per-view noise averages out. Moving cars and pedestrians break this. A point on a passing car is in a different place in every frame. Current systems respond in one of two ways. Some throw moving objects away and keep only the static background. Others handle moving objects with single-frame cues and give up the multi-view benefit. Either way, the moving objects end up as the least complete and noisiest part of the 3D scene: hollow one-sided shells with poorly denoised semantics.
This note proposes the opposite stance. For a rigid moving object, motion is not a nuisance but a free source of viewpoints. A car that drives past a parked sensor rig shows its front, then its side, then its rear. Suppose each moving object's LiDAR points and camera features are registered into the object's own coordinate frame, without box annotations. Seen in that frame, the object is static, and ordinary multi-view distillation applies to it again. The result is a teacher in which moving objects are geometrically completed on their far sides and carry multi-view-denoised foundation features. That teacher then supervises a single-frame LiDAR-camera network that predicts dense 3D semantics, a task known as semantic scene completion.
Core Claim
The claim has two parts. First, the multi-view consistency that makes distillation work, as shown by Neural Feature Fusion Fields, can be restored for rigid dynamic objects by moving into a per-object frame instead of the world frame. Second, the teacher built this way improves a single-frame student mainly on two things: moving-object classes and occluded voxels. Those are where existing methods are weakest. The SemanticKITTI dataset paper reports that segmentation of moving classes "often fell below 20%" (SemanticKITTI).
The new ingredient is not a feature field, a fusion module or a completion decoder, since each of those exists. It is where the aggregation happens: in object-centric frames discovered from foundation masks and rigid registration, applied to LiDAR and camera features together. Whether a learned field beats simple averaging inside that frame is treated as an open ablation, not an assumption.
Relation to Prior Work
Four lines of work set the boundary.
The first line motivates the proposal: methods that resolve the static-scene conflict by removing dynamics. UniSplat predicts a dynamic score per 3D Gaussian, discards moving content, and caches only static Gaussians to fill blind spots, so moving objects stay one-sided. Lift, Splat, Map distills DINOv2 into a bird's-eye grid and inpaints occluded ground, but it explicitly excludes dynamic objects. SCPNet crops moving-object smears out of aggregated labels using external panoptic boxes. That cropping helps dynamic classes by "over 8 points", but it deletes rather than completes those objects. We keep these objects and complete them.
The second line handles dynamics without multi-view aggregation, which constrains our design. CDScene routes moving objects through monocular depth to avoid ghost trails from multi-frame matching. DGD distills CLIP and DINOv2 into Gaussians that a time-conditioned deformation network moves, and it shows that semantic losses sharpen motion tracking. DGD only models visible surfaces, and the paper notes that the deformation network loses track under large inter-frame motion, which is the normal regime for cars at driving speed. Our registration is therefore rigid and explicit, not a continuous deformation field.
The third line supplies the method. Dynamic 3D Scene Analysis by Point Cloud Accumulation shows that registering moving objects as rigid "tubes" into object-centric frames cuts dynamic error from 23.7 cm to 17.3 cm on Waymo. It needs supervised 3D boxes and uses no images. TraPCC accumulates vehicle scans over time, plus a symmetry prior, to build real-world completion targets, but it uses geometry only and covers a single category. We reuse the rigid object-frame idea. We replace box supervision with SAM masks projected onto LiDAR and checked by registration residuals, and we add camera features.
The fourth line supplies the student-side recipe. SCPNet distills from a multi-frame teacher into a single-frame student (+2.8 mIoU). VFG-SSC projects foundation-model predictions into voxels and accumulates them over time with heuristic regularizers against smearing. Our teacher differs from both: SCPNet's teacher is same-modality and closed-set, and VFG-SSC's accumulation happens in the world frame.
Research Program
The pipeline has three stages, applied to logged sequences that contain LiDAR, cameras and ego poses.
Stage 1: segment discovery. For each frame, SAM masks and DINOv2 features are projected onto LiDAR points. Following The Devil is in the Details, projection is done in Cartesian coordinates.
Stage 2: object-frame registration. Masks are linked across frames by feature similarity. Each linked track is registered by robust rigid point-to-point alignment, initialized from the previous transform. A track is accepted as a rigid object if its registration residual stays below a threshold.
This rule has a useful asymmetry. A stopped car registers with an identity motion, so its object frame equals the world frame. Misclassifying slow or stationary objects is therefore harmless here, unlike in UniSplat's memory bank. The dangerous error is the reverse: a moving object treated as static. The residual test targets exactly that error. Tracks that fail the test, which will mostly be pedestrians and cyclists, fall back to per-frame features and are reported separately. We do not claim to complete non-rigid objects.
Stage 3: teacher construction. Inside each accepted object frame, and separately in the world frame for the static remainder, we aggregate two things. The first is LiDAR occupancy, which fills far sides that were seen at other times. The second is camera features rendered to every frame that observed the object. Aggregation is done two ways. The simple way is visibility-weighted averaging of projected features per voxel. The learned way is a small feature field fit per object, as in Feature 3DGS. Object frames are then placed back into each world frame at their registered poses. The result is a per-frame dense voxel target that contains completed occupancy and a foundation feature vector for each voxel.
Student. The student is a single-frame network with SCPNet-style multi-path blocks and early point-pixel fusion. It takes one LiDAR sweep and the synchronized images. It is trained with a binary occupancy loss and a cosine feature-distillation loss on teacher voxels. Optionally, a small labeled fraction adds standard cross-entropy. Closed-set classes are read out by matching voxel features to CLIP text embeddings, and open-vocabulary queries come for free from the same readout.
Baselines
The do-nothing reference projects per-frame 2D foundation predictions onto the current sweep's visible points and leaves everything else empty. It shows how much of any gain comes merely from completion. DGD's comparison suggests this kind of reference can be very weak: 2D LSeg scored 13.90 mIoU against 75.98 for 3D distillation on the Americano scene (DGD). That result comes from a different domain, so we measure the reference ourselves.
The strongest fully supervised method is SCPNet. It reports 36.7 mIoU on the SemanticKITTI completion test set using LiDAR alone, rectified labels and a multi-frame same-modality teacher. It is the strongest competitor because it already contains our two structural moves: a multi-frame teacher and dynamic-aware labels. We retrain it on the same splits, under the same compute budget and hyperparameter sweep size. One variant is LiDAR-only, as published. A second variant has our image-feature input added, so that any gain cannot be attributed simply to adding cameras.
The strongest label-efficient method is VFG-SSC, which reports 11.73 mIoU with 5% labels on SemanticKITTI, against 9.48 for self-training. Its input setting differs from ours, so we rerun it with LiDAR and camera input at 5% and 10% labels. We use the same foundation backbones in both systems, so the comparison isolates world-frame accumulation against object-frame accumulation.
For the label-free setting, we found no reported result on Lacuna for label-free semantic scene completion on SemanticKITTI. Lift, Splat, Map reports 24.1 mIoU label-free, but on CODa, which is a different benchmark. We reimplement its static-world teacher in voxel space as the label-free baseline.
Two ablations isolate the key ingredient. The first is the world-frame teacher: an identical pipeline with object registration switched off. The second is the oracle-box teacher: object frames taken from ground-truth tracks. Together they bound how much is lost by discovering objects without boxes.
What counts as beating a baseline. Beating SCPNet means a significant gain, measured over three seeds, on mIoU restricted to dynamic classes (car, truck, other-vehicle, person, bicyclist, motorcyclist) in occluded voxels, without loss on overall mIoU. Beating VFG-SSC means a gain on overall mIoU at the 5% label budget.
Why not just…?
Why not just project 2D foundation labels onto the LiDAR sweep? This only labels visible points. It cannot reach the far side of an object, and it inherits per-view noise that multi-view distillation removes (Neural Feature Fusion Fields). The do-nothing reference measures this directly. We expect it to remain competitive on visible voxels and to fail on occluded ones.
Why not just accumulate per-object points with off-the-shelf tracks and vote the 2D labels? This is the most serious threat, because it is essentially our Stage 3 with simple averaging and boxes. We do not argue around it. We include it as two cells of our design: the averaging variant of the teacher, and the oracle-box ablation. If averaging matches the learned field, the contribution is the object-frame teacher itself, and the field should be dropped. If oracle boxes greatly outperform discovered tracks, the honest conclusion is that tracking quality, not distillation, is the bottleneck.
Why not just discard dynamic objects and let the student learn them from single frames, as UniSplat and SCPNet effectively do? The world-frame ablation with dynamic voxels masked out implements exactly this. Its weakness should appear on dynamic classes in occluded voxels and nowhere else. If no gap appears there, our premise is wrong.
Why not use a category shape prior such as symmetry instead of temporal views? TraPCC's symmetry prior works for vehicles but does not extend to other categories, and it adds no semantics. We compare against mirror-completion of single-frame vehicle points on the KITTI Tracking completion task. On that task TraPCC reports a mean Chamfer distance of 0.0973 (TraPCC), a measure of average point-to-surface distance where lower is better.
Proposed Experiments
The primary benchmark is SemanticKITTI semantic scene completion. Evaluation uses SCPNet-style rectified labels, because the raw aggregated labels contain the very smears we aim to avoid. We report overall mIoU, dynamic-class mIoU, and the main diagnostic: occluded-voxel IoU, computed only on voxels that the current sweep does not observe. Settings are label-free, 5% labels, 10% labels and full supervision.
Teacher quality is measured separately on Waymo and nuScenes. For geometry, we report the end-point error of registered dynamic points (mean distance from true positions) against the 17.3 cm of the box-supervised accumulation method. For completion, we report vehicle Chamfer distance on KITTI Tracking.
Three decisive checks follow from the claim. First, the object-frame teacher must beat the world-frame teacher on occluded dynamic voxels, and the gap must grow with how much of each object's surface its trajectory reveals. We test this by binning objects by the angular span of viewpoints they present over their track; the claim predicts the gain rises with angular span. Second, gains on static classes should be near zero; a large static gain would mean the improvement comes from something other than object frames. Third, performance on non-rigid classes should match the per-frame fallback, which confirms the method's stated limit instead of hiding it. Failing the first check would falsify the proposal outright.
References
- Neural Feature Fusion Fields: 3D Distillation of Self-Supervised 2D Image Representations
- SemanticKITTI: A Dataset for Semantic Scene Understanding of LiDAR Sequences
- UniSplat: Unified Spatio-Temporal Fusion via 3D Latent Scaffolds for Dynamic Driving Scene Reconstruction
- Lift, Splat, Map: Lifting Foundation Masks for Label-Free Semantic Scene Completion
- SCPNet: Semantic Scene Completion on Point Cloud
- Vision-based 3D semantic scene completion via capture dynamic representations
- DGD: Dynamic 3D Gaussians Distillation
- Dynamic 3D Scene Analysis by Point Cloud Accumulation
- Self-Supervised Point Cloud Completion on Real Traffic Scenes Via Scene-Concerned Bottom-Up Mechanism
- Semi-supervised 3D Semantic Scene Completion with 2D Vision Foundation Model Guidance
- The Devil Is in the Details: Simple Remedies for Image-to-LiDAR Representation Learning
- Feature 3DGS: Supercharging 3D Gaussian Splatting to Enable Distilled Feature Fields