Systems that track a moving camera and build a 3D map from video are now usually split into two halves that barely talk to each other. A vision half estimates where the camera is. A graphics half builds a model of the scene that can render photorealistic new views. The best current example, DROID-Splat, pairs a learned tracker with a 3D Gaussian Splatting map. That map is a cloud of small colored, semi-transparent blobs that a differentiable renderer turns into images. But the tracker never looks at what the map renders. The map is continuously refined from all frames seen so far, so it is arguably the system's best memory of the scene, yet the tracker cannot read it.
We propose closing this loop in one specific way. At each step of the tracker's recurrent update, the current map renders the scene from the current pose estimate. The mismatch between that rendering and the incoming frame, measured in learned feature space, is fed to the recurrent unit as an extra input. The recurrent unit learns when this evidence is trustworthy and when it is not, for example under lighting changes, moving objects, or stale map regions. It then turns that judgment into pose and depth corrections and per-pixel confidence weights. In two sentences: the graphics model's prediction error becomes a sensory input to the vision model's learned optimizer. This lets tracking draw on long-term appearance memory without handing control to brittle raw photometric gradients.
Core Claim
Our claim is that a learned recurrent update operator should read render residuals as inputs rather than minimize them as loss terms. Compared with both modular pipelines and naive photometric bundle adjustment, this should improve two things: tracking accuracy on revisited regions, and robustness under photometric shift. The first gain comes from the map carrying information about frames that have left the tracker's active window. The second comes from the operator learning to down-weight residuals that reflect illumination or dynamics rather than pose error.
Where This Sits
The mechanism comes from the recurrent-optimizer line of work. RAFT replaced coarse-to-fine optical flow with a convolutional gated recurrent unit (ConvGRU, a recurrent cell that updates a spatial hidden state with learned gates). At each step the ConvGRU looks up feature correlations around the current estimate and emits a small correction. DROID-SLAM extended this to many frames and placed a differentiable dense bundle adjustment layer inside the loop. That layer is a Gauss-Newton solve that converts predicted flow corrections into camera pose and per-pixel depth updates. Both systems see only correlations between pairs of images in a bounded keyframe window, and neither ever renders anything. We reuse their operator unchanged except for the new input channel and its effect on the confidence weights.
The evidence for the gap comes from the systems that bolt a renderer on. DROID-Splat runs tracking and Gaussian mapping as parallel modules, and tracking does not use photometric information from the renderer. Flash-Mono goes the other way. Its recurrent network predicts poses and 2D Gaussian surfels in a single feed-forward pass. It reports a finite forgetting horizon that forces sequence partitioning into submaps, and it does not natively handle lighting shifts. We keep an explicit, persistent map and let the recurrent state stay short-lived. Long-term memory lives in the map, and the render residual is the read operation.
The idea that only unexplained prediction error should flow upward comes from PredNet. PredNet is a hierarchical recurrent network in which top-down predictions cancel expected input and only residuals propagate. Its predictions are unconstrained 2D image predictions and blur over long horizons because the model averages plausible futures. Our "prediction" is instead a rendering from an explicit 3D map at an estimated pose. The rendering is sharp wherever the map is well observed, so its residual carries information about pose rather than about averaged futures.
Two results constrain the design. First, per-instance test-time optimization through a generative model is slow (Category Level Object Pose Estimation via Neural Analysis-by-Synthesis; Robust Perception through Analysis by Synthesis). We therefore never backpropagate through the renderer at inference; we only render. Second, TartanAir shows that a day-to-night shift raised ORB-SLAM trajectory error from 0.165 to 9.385. Any method that trusts pixel intensities directly is exposed to exactly this failure. This is why the residual must be computed in learned feature space and gated by the recurrent unit.
The broader motivation is the vision/graphics split documented in the editorial literature (Preface; Editorial). Those pieces argue that analysis methods lack rendering feedback. We treat that argument as a testable engineering hypothesis about one particular feedback path, not as a slogan.
Research Program
The system starts from DROID-Splat's architecture. For every active keyframe edge, the backend's Gaussian map is splatted from the current pose estimate into a 1/8-resolution image. Rendering happens at the tracker's feature resolution, which keeps cost low. The rendered image goes through the same frozen feature encoder as the observed frame. We then form a small local residual volume: for each pixel, feature differences between the observation and the rendering, sampled at a few offsets around the current flow estimate. This is analogous to RAFT's correlation lookup, but against the map rather than another frame. The volume is concatenated with the existing correlation features as ConvGRU input. We also pass the map's per-pixel accumulated opacity and observation count, so the unit can tell a well-observed surface from a hole. The ConvGRU's outputs, flow revisions and per-pixel confidence weights, enter bundle adjustment as before.
Training follows DROID-SLAM on TartanAir, with one change. Each training clip is preceded by a warm-up segment from which a Gaussian map is built and then held fixed. Maps are frozen, so renders are cheap and require no gradient with respect to the Gaussians. To teach the operator when to distrust the render, we corrupt the maps during training in three ways: appearance changes from TartanAir's lighting and weather variants, deleted or displaced Gaussians to mimic dynamic objects, and deliberately stale maps from earlier times. The supervision signal is DROID-SLAM's pose and flow loss. The render residual receives no loss term of its own; it is purely an input. At inference, the backend keeps updating the map as in DROID-Splat, and the frontend renders from the latest snapshot.
We will analyze where the render channel is used. We plot learned confidence against map observation count, against ground-truth illumination change, and against time since a region was last in the keyframe window. The hypothesis predicts high reliance on revisited, well-mapped regions and suppression under lighting change.
Baselines
The naive reference point is unmodified DROID-SLAM, run with identical keyframe window, iteration count and training data. It tells us what the tracker achieves with no map feedback at all.
The strongest existing system is DROID-Splat, since it has the same tracker and the same kind of map and differs only in lacking the feedback path. It reports an Average Trajectory Error of 1.8 cm on TUM-RGBD in Pseudo-RGBD mode, against GO-SLAM at 2.3 cm and GlORIE-SLAM at 2.1 cm. On Replica it reports rendering PSNR of "nearly 40 dB". We will rerun it from released settings on the same GPUs, with the same Metric3D depth prior and the same Gaussian densification. Beating it means lower mean ATE on TUM-RGBD at the same or better PSNR on Replica, with the gap concentrated on sequences that revisit earlier regions.
For monocular speed and accuracy, Flash-Mono reports the lowest ATE across almost all ScanNet sequences at about 12 FPS. It gives no numeric ATE value in the material, so no reported result was found on Lacuna for Flash-Mono ScanNet ATE, and we will run it ourselves. For robustness, DROID-SLAM's 30 of 32 tracked ETH-3D sequences is the bar we must not fall below.
The key ablations isolate the ingredient. The first zeroes the render-residual channel at test time in the trained model. The second replaces it with a render from a shuffled wrong pose, which checks that gains come from pose-informative content. The third feeds pixel-space rather than feature-space residuals. All variants get the same training steps and hyperparameter sweep size.
Why not just…?
Why not just add a photometric term to bundle adjustment? This is the most serious alternative. It amounts to minimizing rendered-versus-observed intensity inside the Gauss-Newton solve, which is how optimization-based Gaussian SLAM refines poses. We will implement it inside DROID-Splat with a tuned weight as a direct competitor. Our prediction is that it helps on static, evenly lit scenes like Replica. It should hurt under TartanAir's lighting variants, where the ORB-SLAM day-night collapse shows that intensity constancy breaks. If the photometric-BA variant matches ours on the lighting-shift splits too, the learned gating adds nothing, and the honest conclusion would be to use the simpler term.
Why not just enlarge the keyframe window or add loop closure? DROID-SLAM already has global backend bundle adjustment and loop closure, and a larger window would let the tracker see older frames directly. Its cost is memory: the paper notes roughly 24 GB for long sequences. A render supplies fused appearance from all past frames at the cost of one splat per edge. We will compare at matched GPU memory, sweeping window size for the baseline. If a larger window at equal memory closes the gap, the claim fails.
Why not just let a feed-forward recurrent network carry the memory, as Flash-Mono does? Flash-Mono documents a forgetting horizon and requires submap partitioning, which suggests a hidden state is a poor long-term store. Our revisit experiments test this directly by measuring error as a function of time since a region was last seen, for both systems.
Proposed Experiments
The decisive experiment uses TartanAir with sequences selected or constructed so that the camera revisits regions after they have left the keyframe window, both with and without an intervening lighting or weather change. The proposal predicts three things relative to DROID-Splat. First, lower ATE on revisits under matched lighting. Second, no degradation under changed lighting. Third, photometric bundle adjustment degrading in the changed-lighting condition. The wrong-pose-render ablation must erase the gains; if it does not, the channel is acting as a generic regularizer rather than as memory.
Secondary evaluations cover TUM-RGBD ATE, where motion blur and rolling shutter stress residual reliability. Replica PSNR checks that feedback does not corrupt the map. ETH-3D success counts guard against new catastrophic failures, and ScanNet monocular ATE and frame rate are compared against Flash-Mono. We also report runtime overhead of rendering per iteration. If rendering overhead exceeds the savings from a smaller keyframe window, that trade-off is part of the result.
References
- DROID-Splat: Combining end-to-end SLAM with 3D Gaussian Splatting
- RAFT: Recurrent All-Pairs Field Transforms for Optical Flow
- DROID-SLAM: Deep Visual SLAM for Monocular, Stereo, and RGB-D Cameras
- Flash-Mono: Feed-Forward Accelerated Gaussian Splatting Monocular SLAM
- Deep Predictive Coding Networks for Video Prediction and Unsupervised Learning
- Category Level Object Pose Estimation via Neural Analysis-by-Synthesis
- Robust Perception through Analysis by Synthesis
- TartanAir: A Dataset to Push the Limits of Visual SLAM
- Preface
- Editorial