Pose-Addressed Generative Retrieval for Camera-Only Place Recognition

Generative retrieval is a way of searching a collection by training a sequence-to-sequence network to write out the identifier of the matching item, rather than comparing the query against a stored vector for every item. It breaks down as collections grow. The usual explanation is that each item's identifier is essentially an arbitrary label the network must memorize, so the parameters needed grow with the number of items. In visual place recognition the task is to recognize where a camera image was taken by matching it against a map built from earlier drives. Here identifiers do not need to be arbitrary: a frame's identity can simply be where the camera was. If identifiers are written as discretized positions along the mapped route, the network no longer memorizes a list. It learns a smooth function from appearance to location, and that function should cost capacity in proportion to the area covered, not the number of frames. The obstacle is obtaining positions without LiDAR or GNSS. Self-supervised depth and ego-motion learning supplies them from raw video: a network learns depth and camera motion by predicting one video frame from its neighbor.

We propose to build a camera-only generative place recognizer whose identifiers come from self-supervised odometry. Frames that see the same place are merged under one identifier when a depth-based warp of one frame reproduces the other. We then test whether this design changes how generative retrieval scales. The concrete prediction can be falsified. At fixed model size, recall should stay flat as more frames are added over the same route, while arbitrary or cluster-based identifiers should degrade.

Core Claim

The scaling failures reported for generative retrieval are failures of memorizing uncorrelated identifiers. Perplexity stops converging as collections grow (TOME). Larger backbones fall behind dual encoders at millions of passages (How Does Generative Retrieval Scale to Millions of Passages?). Parameters grow in step with the corpus (DOGR). One-token-per-item vocabularies explode (GRACE). These results motivate the proposal but come from text and 2D images, where no natural geometry exists.

For places, we claim that identifiers built from a self-supervised trajectory make the memorization load scale with spatial extent. New frames at already-mapped places then need no new weights, only new entries in the prefix tree of valid identifiers. We also claim that photometric warping consistency is a label-free way to decide which frames count as "the same place." This is the part of self-supervised depth estimation that does real work here.

Relation to Prior Work

The closest work is DSI-3D. It maps LiDAR descriptors from LoGG3D-Net to identifiers using a modified GIT decoder. Its variants include positional identifiers linearized with a Hilbert curve, a mapping of 2D coordinates to a 1D string that keeps nearby places nearby. DSI-3D supplies our decoding and identifier-linearization method. It differs from this proposal in three ways: it needs LiDAR, its positional identifiers come from ground-truth coordinates, and it never studies how capacity scales with frame density versus area.

IRGen and GRACE show that image-conditioned generative retrieval can match or beat CLIP-style dense search. IRGen does this with residual-quantized ViT codes trained using class labels. GRACE uses atomic identifiers memorized by a multimodal language model. Both build identifiers from appearance or labels. Neither is available or meaningful for unlabeled driving video, and neither links identifiers to geometry.

On the depth side, Zhou et al. supply the view-synthesis objective. GeoNet supplies the treatment of moving objects that violate the static-scene assumption. DarkSLAM is the nearest systems precedent: self-supervised depth, pose and loop closure on thermal video. Its loop detector is a Siamese network producing continuous vectors that are scanned linearly. We replace that component and keep its depth and pose stack.

Research Program

The system has three stages.

Mapping. A depth and pose network pair is trained self-supervised on the database drive. Its integrated ego-motion gives every frame a position. Drift is tolerable here, because identifiers only need to be locally consistent within the database's own reconstruction, not metrically correct. Scale ambiguity is likewise harmless, and on KITTI stereo training fixes it anyway.

Drift does cause one problem: when the database route crosses itself, the same place gets two positions. We repair this with a covisibility test. For candidate frames $i$ and $j$, we warp $I_j$ into $I_i$ using the predicted depth $D_i$ and relative pose $T_{i\to j}$, following the projection of Zhou et al. The score is

$$s(i,j) = \frac{1}{|\Omega|}\sum_{p\in\Omega} \rho\big(I_i(p), \hat I_{j\to i}(p)\big),$$

where $\Omega$ is the set of pixels that project inside $I_j$ and are not masked as dynamic, $\hat I_{j\to i}$ is the warped image, and $\rho$ is the standard L1-plus-SSIM photometric error. When $s(i,j)$ falls below a threshold, the pair becomes a loop constraint in a pose-graph optimization. After optimization the merged frames share an identifier.

Identifier construction. Optimized positions are quantized to cells, and the cells are linearized with a Hilbert curve, as in DSI-3D. A final token distinguishes heading bins.

Retrieval. An image encoder feeds an autoregressive decoder that emits these tokens under prefix-tree constrained beam search. The decoder is trained on database frames paired with their identifiers and on augmented views. The top-$k$ decoded cells are optionally verified by the same score $s$ against the stored keyframe for each cell.

The analysis centers on two scaling sweeps at fixed parameter count. In the density sweep, the route is held fixed and the frame rate is increased from sparse keyframes to all frames. In the area sweep, density is held fixed and route length grows by concatenating sequences, as DSI-3D does with its combined "sequence 22," and by adding the Largescale Thermal dataset (more than 100,000 images). We plot recall against both axes for each identifier scheme. The central hypothesis predicts a flat curve along density for pose identifiers and a declining curve for arbitrary, hierarchical k-means, and residual-quantized identifiers.

Baselines

Naive reference: exact nearest-neighbor search. We scan exactly over dense global descriptors from the same image encoder, pretrained with the same self-supervised depth stack and fine-tuned with the quadruplet loss that DSI-3D found slightly better than triplet. This matches the continuous-vector loop closure of DarkSLAM. It uses identical images, encoder, and training epochs, and serves as the accuracy ceiling for "no index structure at all."

Strongest practical competitor: graph-based approximate search. We run HNSW over those same descriptors. Its search cost is roughly logarithmic, so the O(N) speed argument made for generative retrieval does not apply to it. We tune it to the same memory budget as our decoder plus prefix tree. Beating it means matching its Hits@1 and F1max while storing a few bytes per frame. On the density sweep, beating it means holding recall at a fixed model size where it must store every added descriptor.

Strongest published generative method: DSI-3D. We run DSI-3D on LiDAR as reported. The file gives only relative results for it: Hilbert-curve DSI-3D exceeds LoGG3D-Net with LSH256 by 0.4% average F1max and 0.1% average Hits@1, and is estimated to beat exact search above about 126,700 descriptors, roughly 8.15 km². No absolute F1max or Hits@1 value is reported on Lacuna, so we will rerun it on the same sequences. A camera-only system within a small margin of LiDAR DSI-3D is the target, not a requirement to beat it.

Oracle: GNSS identifiers. Our identical model, with identifiers taken from GNSS positions instead of self-supervised odometry, bounds how much odometry drift costs.

Ablations. Four ablations isolate the key ingredients:

  • Pose identifiers without covisibility merging, to measure what view-synthesis consistency adds.
  • Covisibility decided by descriptor similarity instead of photometric warping, to show whether geometry, not appearance, is doing the work.
  • Arbitrary identifiers.
  • Hierarchical k-means identifiers.

We will also include an IRGen-style residual quantizer trained without labels, using a contrastive objective in place of IRGen's classification loss, since IRGen's labels do not exist here.

Why not just…?

Why not just use HNSW over a good camera descriptor? This is the real threat. It may match our accuracy, and it indexes new frames trivially. The evidence for generative retrieval in the file is mixed. GRACE beats CLIP on Flickr30K at 68.4% versus 58.4% R@1, and IRGen reports 20.2% higher Precision@10 than fine-tuned CLIP on In-shop Clothes. But DSI-3D's margin over LSH is only 0.4% F1max, and speed crossovers were measured against exact scans, not graph indexes. We therefore do not claim a speed win. The claim we test is that memory and parameters stop growing with frame count. If HNSW at equal memory matches us on both sweeps, the proposal's practical case fails, and we will report that.

Why not just product-quantize or hash the descriptors? Quantization cuts bytes per frame but still stores one code per frame, so storage grows with density exactly as before. Its codes also carry no spatial locality, so nearby places do not share prefixes. The equal-memory comparison on the density sweep settles this directly. DSI-3D's small edge over LSH256 suggests the accuracy gap will be modest, so the argument rests on scaling rather than accuracy.

Why not just regress position directly from the image? A regressor with the same backbone learns the same smooth appearance-to-location map. It is the cleanest test of whether discrete decoding matters. Its weakness is that it outputs one estimate, which averages between visually similar distant places, whereas beam search returns several discrete hypotheses that can be verified. No result on Lacuna addresses this. We include it as a baseline that snaps the regressed position to the nearest database cell, and evaluate it at Hits@1 and at recall@5 after verification.

Why not just use LiDAR or GNSS? When available, they are better, which is why they appear as the DSI-3D reference and the oracle. DarkSLAM shows camera-only self-supervised pipelines keep tracking in thermal and low-light conditions where ORB-SLAM3 fails, and the target platforms often lack both LiDAR and GNSS. The gap to the GNSS oracle quantifies the price of going camera-only.

Proposed Experiments

We will evaluate on KITTI sequences 00, 02, 05, 06, 07 and 08, plus the concatenated sequence 22, using the camera streams and DSI-3D's F1max and Hits@1 protocol. The Largescale Thermal dataset from DarkSLAM covers degraded imaging, and there we additionally report the absolute trajectory error of the full SLAM loop with our retriever substituted for its loop detector. Depth networks are trained on KITTI stereo video in the Zhou et al. setup, with GeoNet-style masking of moving objects.

Three checks decide the proposal:

  • Density sweep. At fixed parameters, pose identifiers must lose no more than a few points of Hits@1 from sparse to full frame rate, while arbitrary and k-means identifiers lose substantially. If pose identifiers also degrade, the core claim is wrong.
  • Merging ablation. Removing photometric covisibility merging must hurt recall on sequences with intra-database revisits, such as 00 and 05, and not elsewhere. That pattern would show the depth-estimation component earns its place.
  • Equal-memory comparison. Against HNSW over the same descriptors, we report accuracy per byte of index and per parameter as the corpus grows. This determines whether the approach has practical value beyond the scaling finding.