Translate the Map Toward Darkness: Hallucination-Free Cold-Start Localization at Night

A robot that has lost track of where it is, or has just been switched on, must find its position and orientation from a single image with no prior guess. This is the "cold start" or "kidnapped robot" problem. Most localization methods only refine a pose they already roughly know, and they fail when there is no starting estimate. Night makes it much worse, because a daytime map and a nighttime photo of the same street look very different. The usual fix is unpaired image-to-image translation: a generator trained on unmatched collections of day and night photos turns the night query into a fake daytime image, which is then matched against the daytime map. We argue this fix runs in the wrong direction. A dark image has lost information, so a generator that "brightens" it has to invent window frames, edges, and textures that it cannot actually see. With no pose prior to veto a bad match, one invented structure that happens to match the map produces a pose that is wrong but looks confident.

We propose to run translation the other way. The well-lit map is converted into the night domain, which only requires the generator to remove or dim information, and the real night query is matched against that synthetic night map without modification. To get past the error floor that retrieval hits when the map has few images, the map is a 3D Gaussian Splatting scene (a 3D representation built from many small colored blobs that can be rendered from any viewpoint). We render it densely across the whole area, translate the renders to night once, offline, and attach depth so that every match becomes a 2D-to-3D correspondence. A standard Perspective-n-Point solver (which recovers camera pose from matches between image points and 3D points) inside RANSAC then returns a global six-degree-of-freedom pose with no initial guess. The scientific question is narrow and testable: in information-asymmetric conditions, does the direction of translation change how often a cold-start localizer is confidently wrong?

Core Claim

The claim is that, when the cold start happens at night, the rate of high-confidence wrong poses is governed by where hallucinated structure enters the pipeline. Translating the query from night to day puts invented structure into the very image that drives hypothesis generation. Translating the map from day to night only darkens content that was observed and is geometrically correct. We predict that the map-side direction yields fewer confidently wrong global poses at the same recall, and that the remaining errors come from a different, measurable source: lamps and lit windows the generator places at the wrong positions.

The research file motivates this from several angles. Long-Term Visual Localization Revisited reports that night queries degrade every matcher it benchmarks. The file's analysis of cycle consistency, the round-trip reconstruction constraint summarized in Generative Adversarial Networks: Recent Developments, notes that this constraint assumes no information is lost between domains, an assumption that breaks when pixels are underexposed. Prior work on the method side has translated only toward the information-rich domain. ToDayGAN turns night queries into day images for retrieval with DenseVLAD (a global image descriptor). Porav et al. add a differentiable SURF keypoint loss and fine-tune on GPS-registered pairs. Mueller et al. translate synthetic renders into realistic photos and report that the generator hallucinates architectural detail at unfamiliar viewpoints.

None of these works asks which side of the matching problem should be translated, and none measures confident failures as a separate quantity from accuracy. Our proposal changes the direction of translation and makes confident failure the primary measurement. That combination is what is new.

Research Program

The map is built from daytime imagery in the style of STDLoc. STDLoc distills learned keypoint features into 3D Gaussians and localizes without a pose prior using PnP followed by dense refinement. We reuse its Gaussian scene and its prior-free PnP stage as the localization backbone. What we change is the appearance domain the map lives in. We render views on a dense grid of positions and headings over the drivable or walkable area, recording rendered depth for each. Every render is then passed through a day-to-night generator trained on unpaired day and night traversals of the same city. Local features and a global retrieval descriptor are extracted from the night-translated renders and stored. Pixels are not stored, so the cost is one offline translation per condition rather than a new map traversal.

At query time, the real night image is used as captured. It retrieves its top candidate renders from the synthetic night database. A frozen modern matcher then matches the query to those renders. The matches are lifted to 3D through the stored depth and solved with PnP inside RANSAC.

The generator needs a guarantee that it does not move geometry. Day-to-night translation has an exploitable property: the true pixel correspondence between a day render and its own night translation is the identity map. We therefore borrow the idea of a frozen-matcher training signal from Patch Your Matcher. That work trains a translator against a correspondence-aware discriminator built on a frozen matcher, but it relies on synthetic cross-modal pairs. In our setting the supervision is free. We penalize any match between the day render and its translation that lands away from the identity location. This yields a geometry-preservation loss in a fully unpaired setting, with no GPS pairs, unlike Porav et al.

We expect the main failure of the map-side direction to be invented light sources. The generator may place lamps or lit windows where none exist. To measure this, we flag translated regions that are much brighter than the day render predicts and count how many RANSAC inliers fall inside them.

The analysis also checks the mechanism behind the claim. For the query-side direction, we measure the fraction of inliers in a wrong-pose solution that originate in regions of the real night query whose raw signal is near the sensor noise floor. Those inliers can only come from structure the generator invented. If the claim holds, query-side wrong poses should be dominated by such inliers. Map-side wrong poses should instead be dominated by invented light sources, or by genuine scene ambiguity.

Baselines

The naive reference point is the same backbone with no translation at all. The real night query is matched directly against the daytime Gaussian renders using the same frozen matcher, retrieval descriptor, and PnP solver. This tells us whether translation helps anything once the matcher is a modern learned one rather than SIFT or SURF.

The strongest existing competitor is query-side translation in the style of ToDayGAN. It is the established, published way to use unpaired translation for night localization. On RobotCar night queries, ToDayGAN reports 52.9% of queries localized within 5 m and 10°, against 13.9% for direct matching. To isolate the direction of translation, we will not simply cite that number. We will retrain the reverse, night-to-day generator ourselves on the identical unpaired day and night training images, with the same architecture, the same identity-correspondence geometry loss applied in the reverse direction, and the same number of GPU-hours and hyperparameter trials. Its output will feed the identical retrieval, matching, and PnP stack. Beating this baseline means a lower rate of confidently wrong poses at matched recall, with recall at least equal. ToDayGAN's reported figure serves as an external sanity check that our reimplementation of the query-side direction is competitive, though our full-pose pipeline is not directly comparable to its retrieval-only output.

A second strong competitor is Porav et al.'s SURF-loss translator. Its reported results on RobotCar are a match success rate rising from 19.5% to 60% and an RMSE (root-mean-square error) falling from 12.74 m to 1.99 m. That pipeline uses GPS-registered fine-tuning pairs and sequential tracking, so we treat it as context rather than as a matched baseline.

An oracle upper bound is a map built from real night traversals, which is the "brute force" multi-condition mapping that Porav et al. argue against. Our method should close most of the gap between no translation and this oracle.

Two ablations isolate the key ingredients. The first removes the identity-correspondence loss, which shows how much geometric drift the generator introduces on its own. The second replaces dense Gaussian renders with only the original database images, which separates the benefit of the translation direction from the benefit of dense viewpoint coverage.

Why not just…?

Why not just use a modern learned matcher on raw night images? This is the most serious threat. If current learned matchers are already robust to darkness, translation in either direction may add nothing. The no-translation baseline settles the question directly. If it matches our method on both recall and confident-failure rate, we will report that translation is unnecessary for modern matchers. The asymmetry analysis would then still measure how much harm query-side translation does.

Why not just apply day-to-night translation as training augmentation for the feature detector, and never translate the map? This is a genuine competitor, and it embodies the same principle of only translating toward darkness. We will run it as a third arm. The same generator augments training of a scene-specific detector in the STDLoc style, and the query is matched against the untranslated day map. If augmentation equals map translation, the contribution reduces to the direction principle, and the cheaper delivery mechanism wins. We would state that outcome plainly.

Why not just normalize the night image with histogram equalization or gamma correction? ToDayGAN already reports that its learned translation beat the histogram-equalized DenseVLAD baseline by more than 2.5 times. More fundamentally, brightening a dark image amplifies sensor noise in the darkest regions, where the useful signal has already been lost. We include equalization of the query as a cheap arm only to confirm that it remains well below learned translation in a modern pipeline.

Why not just use robust estimators or convex relaxations to reject bad matches? Convex relaxations of SE(2) and SE(3) recover the global optimum robustly once correspondences exist. A hallucinated match, however, is not a random outlier. It can be consistent with a coherent but wrong pose, and a robust solver will accept a coherent wrong cluster. We will swap the PnP-RANSAC stage for a robust global solver in both translation directions. If the gap between the directions persists, the problem lies in how hypotheses are generated, not in how they are solved.

Proposed Experiments

The primary benchmarks are RobotCar Seasons night queries and Aachen Day-Night, both evaluated with no GPS or pose prior over the full mapped area. For Aachen Day-Night we found no reported result on Lacuna for any method in the file, so that benchmark will rely entirely on our matched reimplementations.

We report recall at the benchmarks' standard thresholds, including 5 m and 10°. The primary metric, however, is the confident-failure rate: the fraction of queries whose pose is more than 25 m wrong yet carries an inlier count above the threshold that accepts 90% of correct poses on day queries. We also plot a curve of confident-failure rate against recall. On top of these sit the inlier-provenance measurements described in the research program.

The decisive check is as follows. Under matched recall, map-side translation should show a clearly lower confident-failure rate than query-side translation. Query-side failures should trace to inliers from near-noise query regions, and map-side failures should trace to invented light sources. If the two directions tie on confident failures, or if query-side failures do not concentrate in dark regions, the information-asymmetry hypothesis is refuted. We would then report the result as a negative finding about where hallucination actually hurts cold-start localization.