Learning Which Frames a Streaming 3D Reconstructor Should See

Streaming 3D reconstruction models build a map of a scene from video one frame at a time, carrying a running memory forward. They now power robot mapping, and they have a weakness that retrieval-augmented language models share. In retrieval-augmented generation (RAG), one model fetches documents and a second model writes an answer from them. Language models in this setup get worse when given too much loosely relevant text. A streaming reconstructor fed every frame of a busy street likewise absorbs passing pedestrians and cars into its memory, leaves ghostly geometry, and drifts. Today the step that chooses which frames reach the reconstructor is a hand-tuned filter. Nothing about what the reconstructor actually benefits from flows back to it. It is the same "relevant is not the same as useful" mismatch that RAG research has spent years closing.

We propose to train a small frame selector against the downstream quality of a frozen streaming reconstructor's static map, using reinforcement learning. The selector learns to keep frames that add static geometry and skip frames dominated by moving objects or redundant views. The point is to treat the frame stream the way distraction-aware retrieval treats a list of ranked passages. The selector sits upstream of the reconstructor, never edits pixels, and is judged only by what the reconstructor builds. If it works, it should beat motion-error heuristics in two places: on dynamic scenes, and on static objects seen in only a few frames, which outlier-rejection methods tend to erase.

Core Claim

The claim has two parts, and each can be tested on its own. The first is empirical. A recurrent reconstructor such as CUT3R updates a persistent memory with each new frame and reconstructs metric point maps online. We expect it to show an inverted-U curve in the number and kind of frames ingested: map quality rises with more frames up to a point, then falls as dynamic content and accumulated drift corrupt the memory. This mirrors the inverted-U curve that LDAR documents for language models as retrieved context grows. The file notes that CUT3R drifts over long sequences and has no way to reject corrupting frames. Nobody has measured where the peak sits or how dynamic content moves it.

The second part is methodological. A selector trained on the reconstructor's own output quality will find a better subset of frames than error-threshold heuristics at the same frame budget. The comparison point is Adaptive Keyframe Selection, which scores each frame by how much it differs from earlier frames, combining photometric error and a structural-similarity measure after warping. Because that score rises with any visual change, a passing pedestrian triggers a burst of selections. The selector has no way to know whether a frame helps or hurts the reconstruction.

Relation to Prior Work

The closest bridge is Adaptive Keyframe Selection. It places a filter between an RGB-D stream and CUT3R or Spann3r, uses a momentum-adjusted error threshold, and adds a refractory period that blocks selection for a short time after each pick. It motivates our proposal because it shows that frame selection matters for streaming reconstructors. It also constrains our design, because it is the baseline we must beat at matched budget. We differ in three ways: the selection rule is learned rather than set by hand, the training signal is the reconstructor's output, and the selector does not need depth for warping.

The method comes from the RAG side. LDAR trains, with REINFORCE and the generator's accuracy as reward, a light selector that reads only the shape of the retrieval score distribution and chooses a band of passages. BAR-RAG uses GRPO, a policy-gradient method that compares a group of sampled choices against each other, to train an evidence selector against the generator's success. PRCA trains an adapter between a retriever and a black-box model using only the model's output scores. We reuse three things from these papers: the idea of a cheap upstream policy rewarded by a frozen downstream model, group-relative policy-gradient updates, and LDAR's choice to feed the selector compact signals rather than raw content. PRCA also gives a warning we take seriously. Selectors trained against one downstream model may not transfer to another, so we test transfer explicitly.

Distractor-robust radiance-field and Gaussian-splatting methods define what "noise" means in scenes. RobustNeRF fits only the lowest-error half of pixels. NeRF On-the-go predicts per-pixel uncertainty from DINOv2 features, DINOv2 being a self-supervised image feature model. RobustSplat delays the creation of new Gaussians until static geometry settles, and 3DGS-HPC classifies 16×16 patches as static or transient. All four are per-scene optimizers that down-weight pixels inside the loss. None chooses inputs for a feed-forward streaming model.

The file reports a known failure shared by these methods: static objects seen in only one or two views get classified as outliers and erased. That failure motivates our second claim. A selector rewarded on map completeness should learn to keep those rare views rather than discard them. Split4D models moving content explicitly as Gaussians with velocities, but it ingests every frame uniformly. That is complementary to our work, not competing with it.

Research Program

The reconstructor is CUT3R, kept frozen throughout. The selector is a small transformer that sees a sliding window of candidate frames. For each frame it receives cheap features: a downsampled DINOv2 embedding, a dense-flow magnitude histogram, and the reconstructor's own signals from a lightweight "peek" pass, namely its confidence map and how much the memory tokens would change if the frame were written. For each frame it outputs the probability of committing that frame to the reconstructor's memory.

Training samples several selection rollouts per sequence and scores each one by the quality of the final static map. Rollouts are compared within the group, as in GRPO. The reward for a rollout over a $T$-frame sequence is

$$R = -\big(\mathrm{Acc} + \mathrm{Comp}\big) - \lambda \frac{K}{T},$$

where Acc and Comp are the mean distances from reconstructed points to ground-truth static geometry and back, $K$ is the number of frames kept, and $\lambda$ sets the price of each kept frame. Sweeping $\lambda$ traces a quality-versus-budget curve, so comparisons with other selectors happen at equal $K$.

Training data comes from sequences with dynamic content and static ground truth: TartanAir and Waymo Open (both used for training in G-CUT3R), plus synthetic Kubric scenes with controllable distractors. We also build a self-supervised reward variant that needs no ground truth: depth reprojection consistency on held-out static frames, where static regions are identified by agreement across many views. This variant tests whether the method can adapt to new scenes at test time.

The first study measures the inverted-U curve before any learning. We run CUT3R on Bonn and TUM-dynamics sequences with uniformly increasing numbers of frames. We then inject controlled fractions of dynamic-heavy frames in Kubric and record where the peak lies and how it shifts.

Baselines

The naive reference points are feeding every frame to CUT3R and uniform temporal subsampling at the same budget $K$. Feeding everything shows whether selection helps at all. Uniform subsampling shows whether gains come from choosing frames or merely from having fewer of them.

The strongest existing method is Adaptive Keyframe Selection. We will reimplement it with the published combination of photometric and structural error, the momentum threshold, and the refractory period. Its threshold hyperparameters get the same tuning budget on the same training sequences that the learned selector receives. Its single operating point becomes a curve by sweeping the threshold. That paper reports an 82.7% keyframe compression ratio on DTU, but the file contains no reported number on Lacuna for its accuracy or completion on Bonn. We will therefore measure it ourselves. Beating it means lower Acc and Comp error on Bonn and TUM-dynamics at every matched $K$ along the curve, with no loss on static 7-Scenes.

A strong composite baseline combines that heuristic with per-frame dynamic masking. Moving pixels are blanked using a DINOv2 static-or-transient patch classifier in the style of 3DGS-HPC before frames reach CUT3R. This baseline represents "fix the pixels, not the selection."

Two ablations isolate the key ingredient. In the first, the selector is trained with the same architecture and data, but its reward is the heuristic's own error score rather than reconstruction quality. If this matches the full method, the downstream reward is not what matters. In the second, the selector is denied the reconstructor's own signals and sees only image features, which tests whether coupling to the reconstructor's internals is necessary.

An oracle, exhaustive search over frame subsets on short clips using ground truth, gives a ceiling.

Why not just…?

Why not just mask moving pixels and feed every frame? This is the most serious threat. Masking removes ghost geometry from people and cars, but it does not reduce the number of memory updates, so it cannot address drift. Masking also fails on shadows and lighting changes, which RobustSplat explicitly cannot handle. Whether those gaps matter is an empirical question. If the masking-plus-heuristic composite matches our selector on Bonn and TUM-dynamics at equal budget, we will report that the learned selector adds nothing on these benchmarks. We will also stack masking onto our selector to check whether the two are complementary.

Why not just fine-tune CUT3R on dynamic data? Fine-tuning changes the reconstructor's weights, while selection leaves them untouched. G-CUT3R shows that adding inputs to CUT3R helps a lot: it reports about 50% better accuracy on 7-Scenes than unguided CUT3R, and 61% lower absolute trajectory error on Sintel when poses are supplied. None of that shows retraining removes drift over long sequences. We will include a fine-tuned CUT3R given matched GPU-hours as a baseline. We will also test whether our selector, trained on the original CUT3R, still helps the fine-tuned model.

Why not just tune the heuristic thresholds better? The heuristic's score rises with any visual change, so no single threshold can separate a frame showing a new wall from a frame showing a passing pedestrian. Our first ablation, the selector trained to imitate the heuristic, together with the threshold-swept curve settles this directly.

Proposed Experiments

Evaluation uses Bonn RGB-D Dynamic and TUM dynamics as the main dynamic benchmarks, Sintel for camera pose error, 7-Scenes and DTU as static controls where a good selector must not hurt, and NeRF On-the-go sequences for out-of-distribution outdoor clutter. Metrics are accuracy, completion, and normal consistency of the static map, absolute trajectory error, and frames kept.

We add a ghost score: the fraction of reconstructed points that fall inside space swept by ground-truth dynamic objects. We also add a rare-object completion score, computed only on static surfaces visible in three or fewer frames, which targets the failure that outlier-rejection methods share.

Three checks decide the outcome. First, the inverted-U curve must appear in CUT3R on dynamic sequences; if more frames always help, frame selection matters only for compute, and the claim narrows to an efficiency result. Second, the selector must beat both the tuned heuristic and the masking composite on the quality-versus-budget curve on Bonn and TUM-dynamics. Third, rare-object completion must improve over the heuristic, showing the selector keeps sparse static views rather than dropping them as noise.

Transfer is tested by running the CUT3R-trained selector unchanged on G-CUT3R and Spann3r. A large drop would repeat PRCA's lesson that selectors become tied to their downstream model. A small drop would suggest the selector learned something general about which frames help streaming reconstruction.