A neural radiance field stores a whole scene as one continuous function of 3D position. To split it into objects, current methods "lift" 2D masks from image segmenters such as Segment Anything into a 3D field. Each 2D mask comes from one camera. The field is fit to whatever mix of cameras the capture happened to contain. Two problems follow. First, when 2D masks disagree across views, the field settles on the answer favoured by the most frequently seen viewpoints. Those are often not the most reliable ones, because 2D segmenters fail most at grazing angles and occlusions. Second, small or rarely seen objects receive few training rays and get absorbed by their larger neighbours.
Molecular simulation solved a structurally identical problem decades ago. Rare states are sampled deliberately under an artificial bias, and each sample is then multiplied by a known correction weight so that averages still refer to the intended distribution. We propose to do the same for segmentation fields. We will oversample rays toward under-covered objects and boundaries, then reweight every ray's loss so that each surface point is supervised as if it had been seen uniformly from all directions where it is visible. The claim is that this makes 3D segmentation invariant to the capture trajectory and improves small-object recovery, without the boundary damage that unweighted oversampling causes.
Core Claim
Training a segmentation field minimizes an expected per-ray loss $\mathbb{E}_{r \sim q}[\ell(r)]$, where $q$ is the distribution of training rays. Normally $q$ is uniform over the captured pixels, so it inherits the camera trajectory. In noiseless settings $q$ would barely matter, because a flexible field could fit every label. With view-inconsistent 2D masks, however, the same 3D point receives conflicting labels from different rays. The field cannot fit all of them, so its fixed point is a $q$-weighted compromise. The choice of $q$ is therefore part of the answer, not just an optimization detail.
We define the target ray distribution $p$ as follows: each visible surface point $x$ is weighted equally, and the viewing direction $d$ is uniform over the directions from which $x$ is visible. The training loss then becomes
$$ \mathcal{L} = \mathbb{E}_{r \sim q}\big[\, w(r)\, \ell(r) \,\big], \qquad w(r) = \frac{p(x_r, d_r)}{q(x_r, d_r)}, $$
where $x_r$ is the ray's expected surface hit, computed from the field's own depth, and $d_r$ is its direction.
On top of this we add an adaptive bias in the spirit of metadynamics, a simulation method that discourages the system from revisiting already-explored states. The bias is a running coverage count $V_t$ over binned 3D regions and directions, which pushes sampling away from well-covered regions:
$$ q_t(r) \propto q_0(r)\, e^{-V_t(x_r, d_r)/\tau}. $$
Here $q_0$ is the capture distribution and $\tau$ is a temperature that sets how hard sampling is pushed. Because $q_t$ is known up to a constant, the weights $w$ are exact up to normalization.
The ingredient that is new here, as opposed to class-balanced sampling, is the pairing of deliberate oversampling of rare regions with exact correction toward a stated target. A third piece guards against a known failure of this approach: monitoring the effective sample size of the weights, the main diagnostic in enhanced sampling.
Relation to Prior Work
The method comes from molecular simulation. Enhanced Diffusion Sampling steers a diffusion generator toward rare protein states and then recovers unbiased statistics with MBAR, the multistate Bennett acceptance ratio, a self-consistent way to pool samples drawn under several different biases. Its MetaDiff variant accumulates bias in batches, which is the template for our $V_t$. A Kinetic Model of Trp-Cage Folding shows the older WHAM version of the same idea: combining several biased simulation windows into one unbiased estimate. MetaDNS uses metadynamics to keep neural samplers from collapsing onto dominant modes, which is our motivation for treating small objects as "rare states." All of this work targets energy landscapes; none touches multi-view supervision.
The same literature also constrains the design. The file's synthesis of these papers notes that weight variance grows exponentially with bias strength, and that estimates collapse onto a few extreme weights when the biased and target distributions overlap poorly. This is why we cap the bias through $\tau$ and track effective sample size, rather than pushing coverage arbitrarily hard.
On the scene side, three methods already bias ray or point sampling without correcting for it. PCFF steers training rays toward high semantic-entropy boundaries. vMAP samples points around sensor depth. ObSuRF evaluates two points per ray. These choices change which objective is optimized, and the papers do not state which target they are approximating. We make the target explicit.
Two methods supply the host systems we modify, and neither models the ray distribution. DiscoNeRF resolves view-inconsistent masks by Hungarian matching of predicted object slots to 2D masks. GARField learns a scale-conditioned affinity field from SAM masks.
Neural Importance Sampling learns proposal distributions for rendering integrals. That is a variance-reduction problem with a fixed target. Ours is different: we choose the target itself, and the bias exists to reach rare regions.
We do not pursue the file's suggestion of reweighting score distillation in DP-RECON. Score distillation is mode-seeking optimization, not sampling, so there is no ensemble to unbias.
Research Program
We will build a plug-in ray sampler and loss weighting for DiscoNeRF and GARField. Coverage statistics live in a hash grid over surface-point cells crossed with roughly 64 direction bins. Each training ray's surface hit is the depth expected under the current field, refreshed every few hundred steps. The density $q_0$ is estimated by counting the capture's rays per cell and direction bin. The target $p$ assigns equal mass per occupied cell and uniform mass over that cell's visible bins.
When several bias schedules are mixed during training, the per-sample normalizations are unknown. We will pool them with an MBAR-style self-consistent estimate instead of naive per-batch normalization, following the multi-window logic of WHAM and MBAR.
We will log effective sample size per batch. When it falls below a threshold, weights are tempered toward 1. This is reported as an explicit bias–variance knob, not hidden.
The analysis component characterizes when reweighting should matter. Our prediction is that the gain scales with two things together: how much the 2D masks disagree at a point, and how skewed the capture's direction distribution is there. If masks are consistent, $q$ is irrelevant at the fixed point and the method should do nothing.
Baselines
The naive reference point is each host method run unchanged, with uniform sampling over captured pixels. This is what DiscoNeRF and GARField do by default. On Mip-NeRF 360, DiscoNeRF reports 79.24% IoU, against 65.60% for distilled feature fields and 64.12% for Panoptic Lifting with SAM (DiscoNeRF). This is the strongest class-agnostic number in the file and the bar our DiscoNeRF variant must clear. On GARField's bouquet scene, GARField reports 76.0 fine-scale mIoU, versus 17.4 for direct SAM projection (GARField). On ScanNet with inconsistent labels, PCFF reports 0.891 IoU (PCFF). We include it as a reference, but PCFF is a different architecture and we will not claim to beat it unless we reproduce it under identical training.
The strongest competitor on the specific question of ray selection is unweighted biased sampling: PCFF-style entropy-guided boundary sampling and our own coverage bias, both without correction. This is what practitioners already do. All variants share the host's architecture, 2D masks, number of optimization steps, and rays per batch. Each gets the same budget of five hyperparameter settings tuned on a held-out scene. Results are averaged over three seeds.
Two ablations isolate the key ingredients. Static reweighting toward $p$ without adaptive bias tests whether the correction alone explains the gains. Adaptive bias with reweighting but no effective-sample-size tempering tests the safety mechanism.
Beating the baselines means two things. First, a mean IoU gain beyond seed variance on Mip-NeRF 360 and the GARField scenes. Second, and more importantly, a clear gain on the small-object stratum (objects below the 25th percentile of projected area) with no loss on large-object boundary IoU, where we expect unweighted biasing to lose.
Why not just…?
Why not just oversample hard regions without reweighting, as PCFF does? This is the main threat. It may capture most of the small-object gain. Our argument is that it also moves the fixed point toward grazing, boundary-heavy views where 2D masks are least reliable, so large-object boundaries should degrade. The comparison against the unweighted-bias baseline, with boundary IoU reported separately, settles this. If the unweighted version matches ours on every stratum, the reweighting claim fails, and we will report that.
Why not just weight rays by inverse viewpoint density, a standard covariate-shift correction? That is exactly our static ablation, and it may deliver most of the trajectory-invariance benefit. The adaptive bias is justified only if small objects, which static weighting cannot sample more often, improve further. If they do not, the honest conclusion is that the simpler correction suffices, and the stat-mech machinery reduces to the effective-sample-size diagnostic.
Why not just use robust losses or confidence-thresholded masks? Robust losses down-weight outlier labels but still count each viewpoint as often as the capture provides it. A view direction seen 50 times outvotes one seen five times regardless of reliability. Confidence thresholding also discards precisely the rare-object masks we want to keep. We will run a truncated-loss variant of DiscoNeRF under the same budget. The controlled-skew experiment below should show it remains trajectory-dependent.
Why not just project masks and vote? GARField already shows direct projection collapsing to 17.4 mIoU on bouquet versus 76.0 for an optimized field. Voting is included only as the floor.
Proposed Experiments
The decisive experiment is controlled capture skew on Replica, which is used by DFF, PCFF and vMAP and has ground-truth instances. For each scene we render trajectories whose direction distribution ranges from uniform to strongly concentrated on one side of the room. We generate masks with SAM, train DiscoNeRF, and measure instance IoU against ground truth. We found no reported class-agnostic instance IoU on Replica in the file; DFF reports 0.58 semantic mIoU there, which is a different task, so all Replica numbers will be our own reproductions.
The prediction is falsifiable. Uniform-sampling IoU should fall as skew increases, and the reweighted variant should stay flat within seed noise. If both degrade equally, the core claim is wrong.
The second check runs on Mip-NeRF 360 and the five GARField scenes (teatime, bouquet, keyboard, ramen, living_room). We will report stratified IoU by object size and compare against the 79.24% and 76.0 references. Within each stratum we will regress per-object gain on measured mask disagreement times direction skew; the analysis predicts a positive slope.
The third check is the diagnostic. We sweep $\tau$ and plot IoU against batch effective sample size. We expect gains to vanish once effective sample size drops below a few percent of the batch, mirroring the overlap failures known from molecular reweighting. This tells practitioners where the method stops working.
References
- Enhanced Diffusion Sampling: Efficient Rare Event Sampling and Free Energy Calculation with Diffusion Models
- A Kinetic Model of Trp-Cage Folding from Multiple Biased Molecular Dynamics Simulations
- MetaDNS: Enhancing Exploration in Discrete Neural Samplers via Well-Tempered Metadynamics
- Panoptic Compositional Feature Field for Editable Scene Rendering with Network-Inferred Labels via Metric Learning
- vMAP: Vectorised Object Mapping for Neural Field SLAM
- Decomposing 3D Scenes into Objects via Unsupervised Volume Segmentation
- Enforcing View-Consistency in Class-Agnostic 3D Segmentation Fields
- GARField: Group Anything with Radiance Fields
- Neural Importance Sampling
- Decompositional Neural Scene Reconstruction with Generative Diffusion Prior
- Decomposing NeRF for Editing via Feature Field Distillation