Methods that rebuild a 3D scene from video mostly assume the world holds still. When something moves, they either smear it into ghostly clutter or detect it and throw it away. Newer methods split the scene into a static background and a moving foreground, but they decide what is "moving" from low-level signals: where the static model's photometric error is high, or where an event camera fires. An event camera is a sensor that reports per-pixel brightness changes asynchronously, with microsecond timing, instead of full frames. Both signals fail in the same predictable cases. When a moving object stops, the event camera goes silent and the object looks static. When a light switches on, the events and the error look like motion. When a person picks up a bag, one object turns from background into foreground partway through the sequence.
We propose to track each object's motion state over time in an explicit ledger. The ledger is a short symbolic record per object: its identity, a 3D box, and a time-indexed state (static, rigidly moving, articulated, or illumination-only change). A vision-language model proposes the ledger from sparse keyframes. The event stream then checks every proposed state change against physical evidence. The checked ledger decides three things in a 3D Gaussian Splatting reconstruction, which represents a scene as many small colored 3D blobs: which blobs are allowed to deform, which deformation model they use, and where new blobs may be spawned. The claim is that the ledger's persistent object-level memory handles stopped objects, lighting changes, and pick-up events that break per-pixel decomposition, and that it does so without discarding the static map's fidelity.
Core Claim
Current event-plus-RGB dynamic reconstruction separates static from dynamic content from the bottom up and without memory. Event-boosted Deformable 3D Gaussians first trains a static model, marks high-error regions with VGG features, and lifts that mask into 3D. It also jointly estimates the event contrast threshold, which removes "purple haze" artifacts. DEGS adapts an event optical-flow network to each scene and uses a pose network to subtract camera motion, so that event flow can supervise the Gaussian deformation field. Neither method stores the fact that a region was a moving object once its evidence disappears. We claim that a persistent, event-verified object ledger improves reconstruction specifically in stop-and-go and relighting segments. We further claim that the language model matters only for the transition reasoning, meaning which object caused the events and when its role changed, and not for category lookup. The second claim is the one we most expect to fail, and the experiments are built to test it directly.
How the pieces fit
The layout-planning direction supplies the representation. LLM-grounded Video Diffusion showed that GPT-4 can write frame-by-frame box layouts with plausible physics (falling, bouncing, perspective scaling), reaching a 98% layout success rate on its 500-prompt benchmark. SLD closed the loop by having the language model inspect drafts and revise boxes. Both systems plan into a generative void. We keep the idea of an editable, symbolic spatiotemporal plan with iterative correction. The difference is that our plan must agree with sensor measurements and conditions reconstruction rather than synthesis.
The event direction supplies the verifier. Events are dense in time exactly where frames blur. They are also physically structured in a way that distinguishes causes. A moving edge produces spatially compact, polarity-paired events that travel along a direction. A global illumination change produces broad, same-polarity bursts with no coherent displacement. We turn these cues into a cheap per-box test that checks every state transition the ledger proposes. The contrast-threshold estimation from Event-boosted Deformable 3DGS is reused unchanged.
The two static-assumption directions supply the failure being fixed and the constraint. RobustSplat showed that delaying densification, the step that splits and clones blobs where error is high, keeps transient objects from freezing into the background. It also admits that it prunes lighting changes as if they were objects. We adopt its delayed densification. Instead of pruning, we let the ledger send transient content to a dynamic branch. We also allow densification of static blobs only in regions the ledger labels static, or illumination-only with appearance-only updates. MoE-GS showed that no single deformation model suits all motion; it reaches 33.27 dB on N3V by training four experts and learning a router. We replace the learned router with the ledger's motion-type label, which assigns a rigid transform, skinning-style articulation, or a free deformation field to each object. This avoids training every expert everywhere. HOSNeRF handled object pick-up with hand-specified state embeddings and about five days of training on four V100s. The ledger makes those states explicit and automatic.
Research Program
We will build three components and study each one on its own.
The first is the ledger proposer. A vision-language model receives sharp keyframes, recovered from blurred frames through an event double-integral model as in DeblurSplat, together with open-vocabulary detections. It outputs objects with 3D boxes, obtained by lifting detections through the depth map, and a hypothesized state timeline written as code-like entries such as "bag: static until t=3.2, carried by person_1 after". This follows the code-as-layout idea of LayoutGPT.
The second is the event verifier. For each box and time window it computes an edge-motion score, a global-brightness score, and an event-density score. It accepts, rejects, or re-times each transition. Rejections go back to the proposer as text feedback, SLD-style, for at most three rounds.
The third is ledger-conditioned splatting. It builds on the Event-boosted Deformable 3DGS codebase. Deformation is enabled only for blobs inside boxes whose state is non-static at time $t$. Deformation type follows the motion label. Blobs inside a box that later stops keep their dynamic identity with a zero-velocity constraint rather than being reabsorbed into the background, and this is what addresses event silence.
The analysis question is which of three things produces the gains: the persistence itself, the event verification, or the language model's reasoning.
Baselines
The naive reference is Deformable 3DGS trained on RGB alone with no decomposition. Alongside it we run RobustSplat, which represents the "just filter transients" position and is scored only on static-region metrics, since it has no dynamic output.
The strongest existing competitors are Event-boosted Deformable 3D Gaussians and DEGS. Event-boosted Deformable 3DGS is the closest design: event-plus-RGB input, explicit static/dynamic split, and the same Gaussian backbone. It reports an average 2.73 dB PSNR gain over Event-4DGS on its eight synthetic scenes and a 0.78 dB gain from its decomposition alone. The file gives no absolute PSNR for it. DEGS reports 42.15 PSNR on the synthetic Centaur scene and 34.07 PSNR on HS-ERGB Umbrella. For RGB-only object decomposition, Split4D reports 0.893 mIoU on Multi-Human. We run Split4D as a decomposition baseline on scenes where frames are sharp enough for it to work.
All methods get the same input streams, the same total number of optimization iterations, and the same Gaussian count cap. Proposer and verifier time counts against our method's wall-clock budget. Each method's hyperparameters are tuned with the same number of trials on a held-out scene.
Beating the strongest baselines means two things. On the stop-and-go and relighting subsets, we need a statistically consistent gain in dynamic-region and static-region PSNR/LPIPS, plus fewer frozen floaters. On ordinary continuous-motion scenes, including the Umbrella and Centaur scenes where DEGS reports its numbers, we need to be no worse than those reported values.
Three ablations isolate the key ingredients. The rule-ledger ablation keeps everything else fixed but builds the ledger from an open-vocabulary detector, a category-to-motion-type lookup table, and event-activity hysteresis, with no language model. The unverified-ledger ablation uses the language-model ledger without event checking. The memoryless ablation uses event-verified labels recomputed per window without persistence, which is close to the existing bottom-up approach.
Why not just…?
Why not just use an off-the-shelf tracker such as SAM-based video segmentation instead of a planner? Trackers supply identity but not state. They also fail under exactly the blur and contact we target: Split4D reports that tracker-dependent SA4D reaches only 0.592 mIoU on Multi-Human, and HOSNeRF needed manual handling of pick-up transitions. The tracker is still a fair component, and the proposer consumes its detections. The ablation that matters is whether memoryless per-window labeling on top of tracking matches the ledger.
Why not just replace the language model with a category lookup table plus event hysteresis? This is the most serious threat. A table saying "bags are movable, walls are not", combined with "stay dynamic until N seconds of silence", may recover most of the persistence benefit. The rule-ledger ablation tests this head-on. We commit in advance that if it comes within noise of the language-model ledger on every subset, we report the language model as unnecessary and present the event-verified ledger as the contribution. We expect the language model to help only on causal-attribution cases, such as whether the events near a hand belong to the hand or to the bag it grabs, and whether a burst is a lamp switching on or an object moving. We report those cases as a separate subset so any advantage cannot be averaged away.
Why not just prune transients with RobustSplat and keep a static map? Pruning deletes the actors, and RobustSplat itself notes that it mistakes illumination changes for transients and removes valid static geometry. We measure both effects: static-region PSNR during relighting segments, where the verifier's illumination-only label should preserve geometry, and the complete absence of dynamic content in pruning outputs.
Why not just add a learned router as in MoE-GS? MoE-GS trains every expert to convergence before routing, and its router has no memory across time. We compare ledger-assigned motion types against MoE-GS routing on N3V, where MoE-GS reports 33.27 dB. We count it as a success if the ledger matches that figure at substantially lower training cost, even without exceeding it.
Proposed Experiments
We evaluate on the synthetic and real event-based dynamic benchmarks from Event-boosted Deformable 3DGS, the DEGS synthetic scenes and HS-ERGB, and N3V for the motion-type comparison.
Because none of these benchmarks targets stop-and-go motion, we build a stop-and-go suite in two parts. The first part is synthetic: Blender scenes with simulated events and ground-truth masks and states, in which objects pause, are picked up, or experience light switches. The second part is real: HOSNeRF's pick-up scenes (Backpack, Suitcase) replayed on a high-refresh monitor and recorded with an event camera, following the DEGS hardware-in-the-loop protocol, so that real sensor noise is included.
Metrics are PSNR, SSIM, and LPIPS computed separately on static and dynamic regions; decomposition mIoU and state-timeline accuracy against ground truth; the count of frozen floaters, meaning blobs that stay in the background of the static model but are absent from ground-truth geometry; and wall-clock time.
The decisive checks come in three layers. First, the memoryless ablation should degrade specifically in post-stop windows while the ledger does not. If both degrade equally, persistence is not the mechanism. Second, the unverified ledger should hallucinate transitions that the event verifier corrects, visible as timeline errors; this establishes whether sensor grounding is needed at all. Third, the language-model ledger must beat the rule ledger on the causal-attribution subset. That result alone decides whether the layout-planning direction contributes anything beyond an organizing representation.
References
- Event-boosted Deformable 3D Gaussians for Dynamic Scene Reconstruction
- DEGS: Deformable Event-based 3D Gaussian Splatting from RGB and Event Stream
- LLM-grounded Video Diffusion Models
- Self-Correcting LLM-Controlled Diffusion Models
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- MoE-GS: Mixture of Experts for Dynamic Gaussian Splatting
- HOSNeRF: Dynamic Human-Object-Scene Neural Radiance Fields from a Single Video
- DeblurSplat: SfM-free 3D Gaussian Splatting with Event Camera for Robust Deblurring
- LayoutGPT: Compositional Visual Planning and Generation with Large Language Models
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation