Active Frame Selection that Resolves Motion Regimes in Dynamic Scenes

When a 3D model of a moving scene is built from a limited budget of camera frames, the choice of which frames to capture or process matters a great deal. Current methods choose frames that most reduce the model's uncertainty about its own parameters. Moving objects, however, rarely move smoothly. A hand rests, then grabs; a cyclist cruises, then turns. Classical radar tracking handles this with the interacting multiple model (IMM) filter: a small bank of motion models, such as "constant velocity" and "turning," runs in parallel, a probability over which model is currently in charge is updated over time, and the models are continuously re-blended so that idle ones stay accurate. We propose to bring this structure into dynamic 3D reconstruction and to use it to decide where and when to look. Each group of scene elements gets a bank of simple motion experts and a time-varying probability over which expert applies. The system then picks the (camera, timestamp) frames whose images would best reveal which motion regime held, concentrating the budget around regime switches instead of spreading it over predictable motion.

The bet is that, under switching motion, most reconstruction error sits in short windows around regime changes. Those windows are exactly what a regime-aware tracker can predict before seeing any images. Existing active methods for dynamic scenes model motion with one neural network and score frames by parameter uncertainty, so they have no explicit notion of a switch. Passive methods that do mix motion models never choose their data.

Core Claim

Treating the motion regime of each scene region as a latent Markov variable gives two things a monolithic deformation model cannot. The first is a forward-propagated prior over when regime switches are likely, which is available at unobserved timestamps. The second is an image-free score of how well a candidate view would disambiguate competing regime hypotheses. The score comes from rendering the scene under each expert's predicted motion and measuring how distinguishable those renders are from that viewpoint. We claim that a selection rule built on this mode information gain, added to a standard geometry and appearance term, reaches a target rendering quality with fewer frames than parameter-Fisher selection in scenes with abrupt kinematic changes. We also claim the gain shows up mainly in the time windows around switches. The claim is falsifiable in a direct way: if a cheap "pick frames with the most motion" heuristic matches it, the regime structure is not earning its cost.

Relation to Prior Work

The closest work is Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting. It jointly picks camera and timestamp using a Fisher information score over the parameters of a 3D Gaussian Splatting model, a scene representation built from many small colored 3D blobs. In that paper, motion is a single deformation network whose Fisher information is estimated with Hutchinson's trace estimator. We keep its problem setting of choosing (camera, timestamp) pairs from a multi-camera rig under a budget, so the comparison is direct. We replace the single motion network with a switching bank of experts and add a term for regime uncertainty, which its formulation cannot express.

MoE-GS motivates the regime view. It shows that blending heterogeneous deformation experts through a learned router beats any single expert on Neural 3D Video. Its experts are, however, whole dynamic splatting systems trained jointly offline on all frames, with no data selection and high training cost. Our experts are lightweight per-cluster kinematic models, which keeps candidate scoring cheap. Our mixing is temporal and Markov rather than a static spatial router.

Two pieces of tracking work supply the method. The IMM recursion of mixing, parallel filtering, probability update, and combination comes from classical tracking, and its practical value under heterogeneous motion is shown by The IMPTC Dataset, where IMM raised cyclist tracking accuracy from 32.5% to 97.6% MOTA. Differentiable Interacting Multiple Model Particle Filtering shows that the regime models and the transition probabilities can be learned by gradient descent. It does so only on synthetic low-dimensional series with passive measurements and a fixed regime count. We inherit that fixed-count limitation deliberately, and we prune experts that are never used rather than claiming to discover regimes.

The static active-reconstruction line constrains the design. FisherRF shows that rendering-loss curvature can score views without ground-truth images, at about 11 ms per candidate. GO-PRE argues that parameter-space uncertainty is misaligned with pixel error and scores views in image space. Creat3r shows that re-optimizing the model at every step is a bottleneck. Following these lessons, our mode term is computed in image space, needs no new images, and needs no retraining per candidate.

Finally, Active Learning for Control-Oriented Identification of Nonlinear Systems argues that data should be acquired for the downstream objective and that a burn-in phase is needed before targeted acquisition works. We adopt the burn-in phase, using uniformly spaced initial frames. We also take its argument as justification for measuring error in switch windows rather than only on average.

Research Program

The representation starts from a sparse set of uniformly spaced frames. The Gaussians are grouped into a few hundred motion clusters by trajectory similarity. Each cluster carries a bank of $K$ motion experts: constant velocity, constant acceleration, coordinated turn (rotation at a constant rate about an axis), and a small residual network for non-rigid deformation. Each cluster also carries a mode probability over time and a learned transition matrix. Between timestamps we apply the IMM interaction step to expert parameters, so that an expert that has been idle is re-anchored to the current blended estimate instead of drifting. This is the mechanism that keeps the "turn" expert ready when a turn finally happens. Rendering at time $t$ uses the mode-weighted blend of expert-predicted positions.

Frame selection scores each candidate camera $v$ at timestamp $t$ by a geometry and appearance term, for which we reuse the FisherRF diagonal approximation, plus a mode term. For the mode term, let $m_t$ be a cluster's regime at time $t$. Let $\mu_k(t)$ be its predicted probability of regime $k$, propagated forward through the transition matrix from the last observed frame. Let $I_k^v$ be the image the scene would produce from camera $v$ if regime $k$ held. Assuming Gaussian pixel noise with variance $\sigma^2$, we approximate the information a frame gives about the regime with the standard pairwise bound for Gaussian mixtures:

$$ \mathrm{IG}(v,t) \approx H(\mu(t)) + \sum_{k} \mu_k(t) \log \frac{\mu_k(t)}{\sum_{j} \mu_j(t)\, \exp\!\left(-\lVert I_k^v - I_j^v\rVert^2 / 8\sigma^2\right)}, $$

where $H$ is entropy and the sum runs over the $K$ experts. The score is high only when the regime is uncertain at $t$ and the competing hypotheses look different from viewpoint $v$. A view along the direction of motion, where a turn and straight travel render almost identically, scores low. The renders cover only the clusters' screen footprints, so the cost is $K$ partial rasterizations per candidate. We will report the measured cost against FisherRF's per-candidate time.

After each acquired batch, the model is updated for a fixed number of steps. Mode probabilities are updated with the IMM likelihood rule, using photometric residuals as the measurement innovation. The loop then repeats until the budget is spent.

Baselines

The naive reference is uniform temporal sampling with random cameras, at matched frame budget and identical model and training schedule. Next Best View Selections for Semantic and Dynamic 3D Gaussian Splatting reports PSNR 33.60 against 29.66 for random selection on Replica. For its dynamic Neu3D setting, there is no reported numerical result found on Lacuna, so we will re-run it ourselves.

The strongest existing method is that paper's dynamic Fisher selector with its single deformation network. We run it with the same candidate pool, budget, burn-in frames, and per-step training iterations, and give it the same hyperparameter search budget as our method. Beating it means higher held-out PSNR at equal budget. More importantly, beating it means fewer frames to reach a fixed PSNR target, and a clear margin in switch-window PSNR on the scenes with ground-truth switches.

A second strong competitor separates representation from selection. It is our expert-bank model with FisherRF-style parameter Fisher summed over experts and no mode term. If it matches us, the gain comes from the representation alone.

A third baseline is our full model with the mode term computed without temporal structure. Mode probabilities are estimated independently per timestamp, with no transition matrix or interaction step, which reduces the score to plain disagreement among experts. This is the ablation that isolates the IMM ingredient.

As a passive upper reference, we train MoE-GS on all frames. It reports 33.27 dB PSNR on Neural 3D Video and 33.67 dB for its distilled model on Technicolor. Our goal is the fraction of that quality reached at 10–25% of the frames, not to exceed it.

Why not just…?

Why not just pick the frames with the most motion? This is the real threat. Regime switches often coincide with large optical flow, so a flow-magnitude heuristic could capture most of the benefit at near-zero cost. The distinguishing prediction is that fast but steady motion is well explained by a constant-velocity expert, so our method should spend little budget there, while the heuristic spends most of it there. The semi-synthetic benchmark below contains long fast cruising segments with brief turns precisely to separate the two. If the heuristic matches us on switch-window error there, we will report that the regime machinery is unnecessary.

Why not just apply FisherRF to a mixture-of-experts model? Parameter Fisher measures how sensitive the loss is to each parameter. It does not measure whether a view distinguishes between hypotheses. GO-PRE already shows that parameter uncertainty and rendered error diverge in static scenes, and summing Fisher over $K$ experts multiplies cost without targeting switches. The summed-Fisher baseline tests this directly.

Why not just use ensemble disagreement, as in density-aware NeRF ensembles? Disagreement among experts is most of our score. The difference is the Markov prior, which makes regime uncertainty high at unobserved timestamps near predicted switches even before experts visibly disagree. It is also the interaction step, which keeps idle experts calibrated so that their disagreement is meaningful. The no-temporal-structure ablation settles whether these matter. If they do not, the honest conclusion is that a committee suffices.

Proposed Experiments

Experiments run on three data sources. The first is Neural 3D Video/Neu3D and Technicolor multi-camera captures, which match the setting of the strongest baseline. Here, selection picks (camera, timestamp) pairs from the real rig, and evaluation uses held-out cameras at all times.

The second is a semi-synthetic benchmark we build by animating object meshes inside Replica scenes along real pedestrian, cyclist, and vehicle trajectories from IMPTC. This yields realistic switching kinematics with ground-truth regime labels and switch times, and renders from a ring of virtual candidate cameras. We will vary the fraction of time spent in switches and include a single-regime control in which no method should gain.

The third is a stress test in which expert-bank capacity is deliberately mismatched to the true regimes, which probes the fixed-count limitation.

Metrics are PSNR, SSIM, and LPIPS on held-out views, averaged over all times and separately over windows of ±0.5 s around ground-truth switches. We also report frames needed to reach a PSNR target, regime-classification accuracy on the semi-synthetic data, and wall-clock time per candidate and per acquisition step. Everything is reported over five seeds with budget curves from 5% to 30% of frames.

The decisive checks are as follows. On the semi-synthetic benchmark, our selector must beat both the dynamic Fisher baseline and the flow-magnitude heuristic on switch-window PSNR at equal budget. Its acquired timestamps must cluster near true switch onsets, which we measure by the distance distribution from selected timestamps to the nearest switch. Removing the transition matrix and interaction step must shrink the gain. The gain must also vanish on the single-regime control. If the effect is confined to the semi-synthetic data and disappears on Neu3D, that tells us something useful in itself: everyday dynamic captures may not switch regimes sharply enough to reward this structure.