Recycling Transients: Dynamic Gaussian Actors for Sim-to-Real Policy Training

Robots are increasingly trained inside photorealistic copies of real rooms built with 3D Gaussian splatting, a scene representation made of millions of small, semi-transparent colored blobs that can be rendered from any viewpoint at hundreds of frames per second. These copies have a blind spot. They assume nothing in the room moves. When a person walks through the scan video, current pipelines treat that person as noise and erase them. The resulting simulator is empty, so the policy trained in it never practices avoiding a moving person. We propose to keep what gets erased. The people and objects that pass through the capture video can be separated out, reconstructed as reusable animated "actors," and replayed along new paths inside the static copy. This produces moving-obstacle training data that matches the real room's appearance and lighting, at a rendering cost low enough not to slow policy training.

The concrete testbed is vision-based drone navigation in the style of SOUS VIDE. That system builds a Gaussian-splat simulator from a two-to-three-minute handheld video and trains a flight policy that transfers to real hardware with no real-world training. Its simulator assumes a static room, and walking humans appear only at test time, as disturbances the policy has to tolerate. The question we ask is whether a policy that has seen moving humans in simulation actively avoids them in the real world, rather than merely surviving them. We also ask whether Gaussian actors harvested from the scan itself beat cheaper ways of injecting motion.

Core Claim

Static-scene splatting pipelines already detect and isolate transient content as a by-product of cleaning the background. That isolated content is a free source of in-domain dynamic assets. If it is stored in a form that renders in the same single pass as the background, at a fixed per-actor cost, then training against randomized replays of those actors should raise real-world success on tasks that require avoiding moving obstacles. Static-only simulators with test-time adaptation should not get this benefit. The gain should also exceed what comes from pasting 2D cutouts or inserting generic mesh humans, because harvested actors share the capture's lighting, camera, and depth-consistent occlusion with the background.

Research Program

The work has three parts. The first harvests actors, the second renders them within a real-time budget, and the third trains and transfers policies.

Harvesting. We start from the transient-detection machinery of RobustSplat. That method delays the creation of new Gaussians until the static background has settled, and it bootstraps transient masks from coarse to fine resolution. Its purpose is to throw transients away, and the masks it produces are its exit point. We take those masks as our entry point. Within the masked regions we fit a separate set of dynamic Gaussians, anchored in the already-converged background coordinate frame, so the actor's position and scale are fixed relative to the room. Monocular video leaves depth and motion ambiguous, so we borrow two priors. The first is per-frame monocular depth and feature priors, as in StreamSplat, which reconstructs dynamic Gaussians feed-forward from uncalibrated video. The second is velocity-based grouping in the style of Split4D, which assigns each Gaussian a velocity and an identity feature and clusters primitives that move together. The output is one actor per moving entity: a short sequence of Gaussian sets plus a root trajectory.

Monocular harvesting will produce actors that look right only from viewpoints near the original camera. As a controlled fallback, we will also capture a few actors with a small multi-camera rig, so that actor quality and the harvesting idea can be judged separately.

Rendering within budget. Deformation-field methods such as 4D Gaussian Splatting for Real-Time Dynamic Scene Rendering query a small neural network for every Gaussian at every timestamp. For a training simulator that must produce hundreds of thousands of frames, we instead bake each actor into per-keyframe Gaussian sets. At simulation time we place a baked set in the room with a single rigid transform per actor and concatenate it with the background before one rasterization pass. Under this design, render cost should grow roughly linearly in total Gaussian count, because tile rasterization is dominated by per-primitive projection and sorting. That gives a simple budget: the background plus $K$ actors must stay above the frame rate policy training needs. We will measure frame rate against $K$ and against actor size, and we will report where the budget breaks.

Policy training and transfer. The policy architecture follows SOUS VIDE: a lightweight visual encoder, a history network that estimates hidden disturbances from recent states, and a command network running at 20 Hz. A model-predictive expert generates demonstrations, as in SOUS VIDE, but the expert now plans around the actors' known future trajectories. Actors are replayed along randomized paths, speeds, and start times that cross the drone's route. The only change from SOUS VIDE is therefore what the simulator shows and what the expert avoids. The policy class and the onboard compute are unchanged.

Baselines

The naive reference point is SOUS VIDE trained exactly as published. That means a static splat simulator, with humans encountered only at deployment. The paper reports 93.3% success in a cluttered workshop with obstacles 1.0 m apart and 100% success with 0.17 m trajectory tracking error on standard trajectories (SOUS VIDE). It also states that flight remained stable with dynamic human obstacles present, but no reported result found on Lacuna for a success rate specific to that condition. We will measure that number ourselves and treat it as the bar. The baseline is retrained from the same scan, with the same number of expert demonstrations and the same training steps.

The strongest competitor on the visual side is compositing 2D human cutouts into rendered frames with randomized placement. This is cheap domain randomization and requires no 3D reconstruction. A second strong competitor inserts generic animated human meshes rendered by a conventional rasterizer, with occlusion against the splat depth. This mirrors how SplatSim moves rigid object clusters with physics-engine transforms, and how driving simulators like DrivingGaussian composite separately reconstructed dynamic objects into a static background. Both competitors receive the same expert, demonstration count, actor trajectory distribution, and tuning budget. Only the obstacle's appearance pipeline differs.

The ablation that isolates the key ingredient keeps harvested Gaussian actors but replays them only along their recorded trajectories. This separates "in-domain appearance" from "trajectory randomization." A second ablation removes depth-consistent occlusion by drawing actors on top of everything, to test whether 3D placement matters. Beating the baselines means two things on real dynamic-obstacle flights. First, a statistically separated higher success rate and larger minimum human-drone distance than both static SOUS VIDE and the strongest of the two compositing baselines. Second, no regression on the static-course success and tracking error reported above.

Why not just…?

Why not just rely on the history network, as SOUS VIDE already does? This is a real threat. SOUS VIDE shows that test-time adaptation absorbs wind, payload changes, and the presence of people. Adaptation from past states, however, estimates disturbances to the drone's own dynamics. It gives the policy no learned reason to change its path when a person steps into it. Our dynamic-obstacle trials are designed to separate tolerance from avoidance by placing people on the planned path at controlled times. If static SOUS VIDE matches our success there, the proposal's main claim is falsified, and we will report that.

Why not just paste 2D cutouts as augmentation? Cutouts teach the encoder that people exist, but they have no consistent depth, scale against the room, or parallax across consecutive frames. Avoidance depends on exactly those cues. The cutout baseline above tests this directly. If it matches harvested actors, the 3D reconstruction is unnecessary, and we would recommend cutouts instead.

Why not just insert mesh avatars? This is the standard compositing route, and it controls geometry well. It reintroduces the hand-built appearance gap that splat simulators exist to remove, including mismatched lighting and textures. The appearance problem documented in sim-to-real work is the reason SplatSim and SOUS VIDE moved to splats in the first place. The mesh baseline, run with matched trajectories, measures how large that gap is for moving obstacles specifically. If the gap is small, the harvesting step is not worth its cost.

Why not just scan the room several times with people in different places? Static snapshots of people never move during training. They teach static-obstacle avoidance, which SOUS VIDE already handles at 93.3%. We include a frozen-actor control, which places harvested actors at random poses with no motion, to test whether motion itself matters.

Proposed Experiments

Reconstruction quality is checked first. On NeRF On-the-go, where RobustSplat reports gains over SpotLessSplats and WildGaussians, our harvesting must leave background quality (PSNR, SSIM, LPIPS) no worse than RobustSplat's transient-free output. No reported result found on Lacuna for RobustSplat's exact values, so we rerun it. For actor fidelity, we evaluate multi-rig captures on held-out cameras, and we report fidelity as a function of angular distance from the capture viewpoint for monocular actors.

The rendering budget is checked next. We plot simulator frame rate against number of actors on the same hardware class used for training, with 130 FPS for static FiGS as the anchor reported by SOUS VIDE. We also report 4D-GS's 82 FPS at 800×800 on an RTX 3090 (4D-GS) and StreamSplat's 49 ms per frame (StreamSplat) as reference points for neural deformation and feed-forward alternatives.

The decisive experiment uses real flights in two scanned rooms. Each condition gets a fixed number of trials. Conditions are no humans, a human crossing the path at a scripted time, and a human walking freely, with all methods flying identical scripts. The metrics are success, collision rate, minimum separation, and tracking error. The proposal succeeds if harvested actors with randomized trajectories beat static SOUS VIDE and the best compositing baseline on the crossing condition while matching them on the empty course. If monocular actors fail but rig-captured actors succeed, the conclusion narrows: dynamic Gaussian actors help, but harvesting them from casual scans does not yet work.

Shadows cast by the actors are not modeled. This matches the limitation Split4D notes for object removal. We will report this as a known source of residual gap rather than claim to solve it.