Methods that build 3D scenes from casual photos have to discard "transients", meaning people, cars or hands that pass through only some of the images. Most of them decide what to discard with one rule: anything whose color disagrees across views is probably a moving object, so mask it out. Glass breaks this rule. A stationary wine glass looks different from every viewpoint because it bends the background behind it into a new pattern each time the camera moves. Thin structures such as rims, stems and wires add a second problem. They cover so few pixels that they look like noise to filters that judge points by how densely they are surrounded. The result is that the objects robots most need to grasp, and the details that make a reconstruction usable, are the parts most likely to be erased by the step meant to clean the scene.
We propose to replace "does the color disagree?" with "is the disagreement explained by where the camera looked from, or by when it looked?" A static refractive surface point changes color as a function of viewing direction. Two images taken from nearly the same direction should agree even if they were captured minutes apart. A transient changes color as a function of time, so two images from the same direction at different times disagree. We turn this into a per-primitive cross-validation test inside 3D Gaussian Splatting, a scene representation built from many small colored 3D blobs that are fit to images. Primitives whose color is predictable from viewing direction alone are kept as static and given more view-dependent capacity. Primitives that are only predictable within a time window are treated as transients. Because the test is run on each 3D primitive rather than on pixel neighborhoods, it does not penalize a structure for being thin.
Core Claim
The leading transient-removal pipeline, RobustSplat, delays Gaussian growth until the static geometry has settled and predicts transient masks coarse-to-fine. It treats high photometric residual (the color error left over after fitting) as evidence of motion. The research file's analysis points out that this assumption should flag stationary transparent objects as distractors, but no one has measured it. The same objection applies to MS-GS and to the motion-consistency decomposition of Split4D, which assumes surface features translate smoothly. Our claim has two parts. First, residual magnitude is the wrong statistic for deciding what is transient. Second, the right statistic is whether a primitive's color can be predicted from viewing direction across time. A filter built on that statistic should keep static glass and thin rims while still removing true transients, including transparent ones such as a glass carried through the scene.
For a primitive $i$, we split its observations into time blocks. For each held-out observation $k$, we predict its color $\hat c_{ik}$ by kernel regression over the primitive's observations from other time blocks with nearby viewing directions. We compare this error to the within-block error $e_i^{\text{in}}$:
$$\rho_i = \frac{\operatorname{mean}_k \lVert c_{ik} - \hat c_{ik} \rVert^2}{e_i^{\text{in}} + \epsilon},$$
Here $c_{ik}$ is the observed color and $\epsilon$ is a small constant. A static primitive, even a strongly view-dependent one, should give $\rho_i$ near one. A transient should give a large $\rho_i$. Kernel regression is used instead of low-order spherical harmonics because refraction makes color vary sharply with direction. A nonparametric interpolator only assumes that nearby directions give similar colors.
The test is honest about its blind spot. It needs viewing directions that overlap across time blocks. When a held-out view's direction lies outside the directions seen in other blocks, the primitive is marked undecided and falls back to RobustSplat's default behavior. We report how much of each scene falls into this undecided regime.
Research Program
The system is RobustSplat with its transient-mask decision replaced by the $\rho_i$ test. We keep its delayed densification, which the file reports is necessary because turning densification off entirely over-smooths fine detail. The test runs on the primitives that exist after the delayed-growth phase. Primitives classified as transient lose supervision from the time blocks where they fail. Primitives classified as static but view-dependent get a small view-conditioned color head instead of the default spherical-harmonic color. This is the step that lets glass be modeled rather than masked.
To protect thin structures, we audit every density-based pruning step in the pipeline. This includes isolation-style spatial filters of the kind used in EAO-SLAM and percentile residual cuts like those in U-RED. Both reject points for being sparse or high-residual, which is exactly how a rim or wire appears. In our pipeline, those steps are gated so they cannot remove a primitive that has passed the static test.
Geometry is extracted for evaluation in two ways. The first is the standard Gaussian depth render. The second, for thin parts, applies the decoupled distance-and-sign extraction of Metric–Phase Fields to the retained primitive centers. That paper reconstructs open thin shells from unoriented points but assumes clean, static input. We supply input that is meant to be cleaner on exactly the points it needs.
The work has three parts. The first is a diagnostic study measuring how often existing transient filters delete static transparent and thin objects. The second is building the filter and its fallback. The third is a new evaluation set, because no benchmark in the file combines static glass, thin parts and real transients with geometric ground truth.
Baselines
The naive reference point is plain 3D Gaussian Splatting with no transient handling. It shows how badly transients corrupt the scene, and it also gives the best possible treatment of static glass, since nothing is masked. Every other method should be judged against both of these edges.
The strongest existing competitor is RobustSplat. It is the closest prior work: it already frames transient removal as consistency filtering during Gaussian optimization, and our method is a targeted change to its masking decision. The file reports that it beats SpotLessSplats, WildGaussians and T-3DGS on NeRF On-the-go and RobustNeRF, but no reported result was found on Lacuna for RobustSplat's PSNR, SSIM or LPIPS values on those datasets. We will therefore rerun it from the released configuration on every scene, using the same images, poses, iteration count and GPU budget as our method. Its mask threshold will be tuned on the same validation scenes we use to tune the $\rho_i$ cutoff.
Split4D serves as the decomposition-style competitor. Its reported number is 0.893 mean IoU (intersection-over-union, averaged over objects) on Multi-Human against 0.593 for SA4D. That result is on opaque scenes, so it does not predict performance on glass.
For geometry of thin and semi-transparent static objects, we compare against αSurf. It decouples a level-set geometry field from an opacity field and reports an average Chamfer distance of 0.284 on Thin Blender, against 0.454 for the next best method. It assumes a fully static scene, so it is an upper reference on clean scenes and a victim on scenes with transients.
The ablation that isolates our key ingredient keeps everything identical except the classification statistic. One variant thresholds per-primitive residual magnitude. The other uses $\rho_i$. If the residual-magnitude variant matches $\rho_i$, the claim fails.
Beating RobustSplat means three things on our evaluation set. First, a clearly lower rate of static transparent pixels masked as transient. Second, lower depth error and boundary-band Chamfer distance on glass and thin parts. Third, no worse transient recall and no worse PSNR on opaque background regions.
Why not just…?
Why not just give RobustSplat more view-dependent capacity, such as higher-order color or a view-conditioned head everywhere? This is the most serious threat. If extra capacity lowers the glass residual below the mask threshold, the problem goes away without any new test. The danger in the other direction is that the same capacity also lets the model explain transients as "view-dependent" color, because nothing separates viewpoint from time. We run this variant directly, with the capacity matched to our view-conditioned head. If it keeps glass and still removes transients as well as our method, we will report that the fix is capacity, not classification.
Why not just segment transparent objects with a 2D network and exempt them from masking? This fails when the transparent object is itself the transient, such as a glass being carried, which is a case our test handles. It also relies on 2D segmentation that the file reports is fragile on glass. The Transparent Object Tracking Benchmark found that generic trackers follow the background visible through the glass rather than the glass boundary. We include a mask-exemption baseline using ground-truth transparent masks, which gives an optimistic version of this idea. We report it separately on scenes with static and with moving glass.
Why not just scan an empty workspace first, as Residual-NeRF does? Residual-NeRF freezes a field of the empty scene and learns only what is added, reporting 46.1% lower depth RMSE than NeRF and Dex-NeRF baselines. However, it needs an empty pre-scan and assumes nothing moves during each capture. Evo-NeRF likewise handles only step-wise changes between grasps. Neither assumption holds for casual capture with people walking through. Where an empty scan is available, we report Residual-NeRF as a reference.
Proposed Experiments
The first check is diagnostic and cheap. We run RobustSplat on static sequences from ClearPose, which provides real RGB-D sequences with SLAM camera trajectories, object poses and CAD meshes for 63 transparent objects. Because the scenes are static, every pixel masked as transient is an error. Rendering the CAD meshes at the annotated poses gives exact transparent-region masks and ground-truth depth. If RobustSplat masks little of the glass, the motivating gap is smaller than the file suggests, and we would redirect effort toward the thin-structure pruning problem instead.
The second experiment uses a synthetic set rendered in Blender Cycles, following the domain-randomized setup of ClearGrasp. It includes path-traced glassware, wire racks and stemmed glasses, with walking people and moving opaque and transparent props. Every scene comes in a clean version and a version with transients, so we know exactly which primitives should survive. Metrics are false-transient rate on static glass and thin parts, transient recall, held-out-view PSNR split by region type, depth RMSE in transparent regions, and Chamfer distance restricted to a band around thin parts.
The third experiment uses real captures of ClearPose-style objects with a person moving through the scene. Geometry is scored against CAD meshes registered at the start of each capture. We also report the undecided fraction from the fallback rule as a function of how densely the camera revisits similar viewing directions.
The decisive comparisons are the residual-magnitude ablation and the capacity-matched RobustSplat. If neither closes the gap on false-transient rate while transient recall holds, the claim stands: viewpoint-versus-time predictability is what separates glass from motion.
References
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- MS-GS: Multi-Appearance Sparse-View 3D Gaussian Splatting in the Wild
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- EAO-SLAM: Monocular Semi-Dense Object SLAM Based on Ensemble Data Association
- U-RED: Unsupervised 3D Shape Retrieval and Deformation for Partial Point Clouds
- Metric—Phase Fields: Decoupling Distance and Sign for Thin-Structure Reconstruction from Unoriented Point Clouds
- αSurf: Implicit Surface Reconstruction for Semi-Transparent and Thin Objects with Decoupled Geometry and Opacity
- Transparent Object Tracking Benchmark
- Residual-NeRF: Learning Residual NeRFs for Transparent Object Manipulation
- Evo-NeRF: Evolving NeRF for Sequential Robot Grasping of Transparent Objects
- ClearPose: Large-scale Transparent Object Dataset and Benchmark
- Clear Grasp: 3D Shape Estimation of Transparent Objects for Manipulation