A camera driven down a long city street slowly loses track of where it is. Each frame's position is estimated relative to the previous one, so small errors compound, and straight streets come out bent. Aerial photographs taken looking straight down (nadir imagery) offer a fix, because they show the true layout of road markings, curbs, and building footprints. The ground track can be pinned to that layout. The trouble is that the features both views share sit at street level, which is exactly where cars, buses, and pedestrians stand. Correction therefore fails in the busiest streets, which are also the places where satellite positioning is worst. The reverse problem also holds. A drifting trajectory makes permanent structure look inconsistent from frame to frame, so filters designed to remove moving objects start deleting real walls and road paint.
We propose to treat these two problems as one. The ground reconstruction gives each 3D element a learned persistence score: how likely it is to be a permanent part of the scene. Three independent sources vote on that score. The first is the ground video itself over time. The second is the aerial image, which was captured at a different moment with different traffic. The third is the current corrected trajectory. The permanent part of the reconstructed road surface is accumulated over many frames into a top-down road picture with the traffic removed, and that picture, not individual frames, is aligned to the aerial image to correct drift. The claim is that drift correction then stays accurate as traffic density rises, while existing aerial-anchoring pipelines degrade.
Core Claim
Existing aerial–ground trajectory correction assumes the shared features are visible and static. The closest work, CORRECTION OF MOBILE MAPPING TRAJECTORIES IN GNSS-DENIED ENVIRONMENTS USING AERIAL NADIR AND AERIAL OBLIQUE IMAGES, matches road markings frame by frame to orthophotos and facades to oblique aerial images, then runs a global bundle adjustment (a joint refinement of all camera poses and 3D points). It reports 11–18 cm horizontal error on Rotterdam but has no model of occluders. Drift-free Visual SLAM using Digital Twins aligns sparse tracked points to a city mesh and fails when dynamic objects enter view or the mesh is out of date. On the other side, transient-removal methods such as RobustSplat and 3D StreetUnveiler assume poses are already correct.
Our contribution has two parts that are specific and testable. First, the anchoring signal is a time-aggregated, transient-free top-down rendering of the road surface. This recovers paint and curbs that are only ever visible in gaps between cars. Second, the aerial image acts as a cross-time witness for persistence. Something present in the ground video but absent from the aerial image is probably transient. Something present in both may be permanent or may be a long-parked car, and the ground video's temporal evidence breaks that tie.
Research Program
The ground-side representation is a 3D Gaussian Splatting model, which represents the scene as many small colored, semi-transparent ellipsoids that are rendered by projection. We build it incrementally over sliding windows of the trajectory. Each Gaussian $g$ carries a persistence score $p_g \in [0,1]$.
Densification, the step where the method adds new Gaussians in regions of high rendering error, is gated by persistence. We borrow the delayed-growth idea from RobustSplat so that high-error transient regions do not spawn floating Gaussians before $p_g$ has settled.
The persistence score combines three log-odds terms. The first measures how consistently the Gaussian is photometrically supported across revisits at different times within the drive. This is a per-element analogue of the trimmed residual statistics in RobustNeRF, applied to elements rather than pixels. The second compares the Gaussian's top-down rendered appearance with the aerial orthophoto at its current corrected location. The third is a weak semantic prior from an off-the-shelf segmenter. We keep this prior deliberately weak, because Split4D shows how missed or mislabeled 2D masks propagate into 3D.
For each window, the drift correction $\Delta T$ is a planar rigid-plus-yaw correction of the window's poses. It is the drift mode that bends straight streets and that nadir imagery can observe. It is solved by
$$\Delta T^{*} = \arg\min_{\Delta T} \sum_{u} w(u)\, \rho\big(O(u) - R_{\Delta T}(u)\big),$$
where $u$ ranges over orthophoto pixels, $O$ is the aerial orthophoto after photometric normalization, $R_{\Delta T}$ is the top-down rendering of the ground model under the corrected poses, and $w(u)$ is the rendered accumulated persistence-weighted opacity at $u$. Only ground surface observed as permanent contributes to the match. The function $\rho$ is a robust penalty.
The persistence update and the drift correction alternate. Each window's correction enters a pose graph as a soft absolute constraint, alongside the relative odometry constraints between consecutive frames. Oblique facade constraints, handled in the manner of the Rotterdam pipeline, are added where available to constrain along-track translation on streets whose markings repeat. We will also analyze observability, meaning which drift directions each anchor type actually constrains. This tells us when a window's anchor should be down-weighted, for example a straight, unmarked road that constrains lateral position and yaw but not along-track position.
Baselines
The naive reference is ground visual odometry or incremental structure-from-motion with no aerial anchoring. We use COLMAP-style incremental reconstruction as described in Structure-from-Motion Revisited, and also STream3R as a modern streaming alternative without loop closure. These establish how much drift exists and how it scales with trajectory length. Our method should cut the endpoint drift of both.
The strongest existing competitor is the Rotterdam aerial-correction pipeline CORRECTION OF MOBILE MAPPING TRAJECTORIES.... It addresses exactly this task with the same nadir-plus-oblique inputs and reports 2D RMSE of 11–18 cm and 3D RMSE of 15–21 cm against 34 surveyed control points. We will reimplement it on our data with the same odometry front end, the same aerial imagery, and an equal hyperparameter tuning budget on a held-out trajectory. Beating it means matching its error on low-traffic segments and showing clearly lower error, plus higher anchor yield, on high-traffic segments.
We also compare against digital-twin registration Drift-free Visual SLAM using Digital Twins, which reports about a 31–32% reduction in absolute trajectory error over VINS-Fusion and ORB-SLAM3. It runs on the same odometry, using a mesh derived from the aerial data in place of Google 3D Tiles.
For the reconstruction itself, the reference points are Horizon-GS Horizon-GS and RobustSplat, both run on the corrected poses. Horizon-GS is a static aerial–ground Gaussian model that reports a synthetic street-view PSNR of 23.80 against 21.81 for plain 3DGS. RobustSplat is a transient-free single-scale model.
The key ablation is "segment, then aggregate, then match." It uses the same aggregated top-down road rendering and pose graph, but replaces learned persistence with hard semantic masks. A second ablation removes the aerial-witness term from the persistence score.
Why not just…?
Why not just mask cars and people with a segmenter and run the existing correction pipeline? This is the real threat, and the "segment, then aggregate, then match" ablation exists to test it. The literature gives reasons to expect it to fall short. 3D StreetUnveiler found that 2D masking over-prunes background that is visible around occluders, and Split4D documents missed masks propagating into 3D. Semantic classes also miss vehicle shadows on paint, construction barriers, and changes since the aerial flight. If the ablation matches the full method, we will report honestly that the gain comes from time-aggregated rendering rather than learned persistence, and narrow the claim accordingly.
Why not just use robust kernels or RANSAC (repeated fitting on random subsets to reject outliers) in the aerial–ground matching? Robust rejection needs the static content to dominate. In dense traffic the static content is a minority in any single frame, which is the failure mode of trimmed losses discussed around RobustNeRF. Our baseline already uses a robust penalty $\rho$, so the experiment isolates the effect of aggregating over time before matching.
Why not just use RTK-GPS and LiDAR? Both Drift-free Visual SLAM using Digital Twins and the Rotterdam work report GNSS multipath degradation in urban canyons, and LiDAR-reliant systems like Street Gaussians exclude camera-only platforms. We treat RTK as ground truth where it is reliable, not as a competitor.
Proposed Experiments
The controlled study uses MatrixCity MatrixCity, which provides synchronized aerial and street imagery, ground-truth poses and depth, and adjustable dynamic elements. We sweep traffic density from empty to saturated, inject odometry noise matched to measured real drift, and render the aerial epoch with different traffic than the ground epoch. The metrics are:
- absolute trajectory error;
- drift per 100 m;
- straightness error on straight streets;
- anchor yield, the fraction of street length with an accepted correction;
- persistence F1 against ground-truth dynamic labels, including parked-but-moved cars;
- street-view PSNR.
For reference, MatrixCity's own paper reports street-view PSNR of 23.2 for MipNeRF-360.
Real-world validation uses the Rotterdam mobile mapping trajectories and their control points. We additionally split segments by traffic density, which we annotate.
Transient-separation quality on the ground side alone is checked on Waymo, against 3D StreetUnveiler's reported LPIPS of 0.216. The file lists no aerial imagery for Waymo, so this test covers removal, not anchoring.
The decisive check is the curve of trajectory error against traffic density. Baselines that match frame by frame should degrade as density rises, and our method should stay roughly flat. The failure condition is a curve parallel to the segmentation ablation, which would mean learned persistence adds nothing. A second check is the deliberate staleness test, with buildings and parked cars changed between the aerial and ground epochs. The aerial-witness term should lower the scores of changed structures instead of pulling the trajectory toward an outdated map, which is the failure the digital-twin approach acknowledges.
References
- CORRECTION OF MOBILE MAPPING TRAJECTORIES IN GNSS-DENIED ENVIRONMENTS USING AERIAL NADIR AND AERIAL OBLIQUE IMAGES
- Drift-Free Visual SLAM Using Digital Twins
- RobustSplat: Decoupling Densification and Dynamics for Transient-Free 3DGS
- 3D StreetUnveiler with Semantic-aware 2DGS - a simple baseline
- RobustNeRF: Ignoring Distractors with Robust Losses
- Split4D: Decomposed 4D Scene Reconstruction Without Video Segmentation
- Structure-from-Motion Revisited
- STream3R: Scalable Sequential 3D Reconstruction with Causal Transformer
- Horizon-GS: Unified 3D Gaussian Splatting for Large-Scale Aerial-to-Ground Scenes
- Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting
- MatrixCity: A Large-scale City Dataset for City-scale Neural Rendering and Beyond