Abstract
Recalling the most relevant visual memories for localisation—or understanding a priori the likely outcome of localisation effort against a particular visual memory—is essential for efficient and robust visual navigation. Solutions to this problem should be independent of ground-truth performance appraisal, as such data is unavailable at run-time, and should ideally rely on generalisable environmental observations. We propose applying the recently developed "Visual DNA" (vdna) as a highly scalable tool for comparing datasets of images—specifically, sequences of map and live experiences. By comparing neuron-wise activation statistics between live images and multiple previously recorded past experiences, we can predict localisation performance across significant seasonal and diurnal shifts. Our approach, validated on the Nordland and University Parks datasets, demonstrates an excellent ability to rank candidate experiences based on actual localisation performance.
Introduction
For an autonomous robot to navigate effectively, it must compare its current visual input to a stored map of the environment. However, the world is dynamic: a park looks fundamentally different during a summer sunset than on a snowy winter morning. This phenomenon, known as domain shift, presents a significant challenge for visual place recognition (VPR).
If a robot attempts to localise against every map it has ever recorded (e.g., Summer, Winter, Day, Night), it incurs massive computational costs. Conversely, picking the wrong map leads to localisation failure. The challenge lies in experience selection: how can a robot rank its stored experiences a priori and select the one that best matches current conditions without access to ground truth (like GPS) to verify its choice?
Methodology: Visual DNA
We address this by using Visual DNA (vdna) to measure the similarity between datasets. This method moves beyond individual image features to look at the broader statistical "fingerprint" of a sequence of images.
- The Fingerprint: A sequence of images (the "warmup frames") is passed through a pre-trained deep neural network, such as ResNet or a Vision Transformer. We record the distribution (histograms) of activations for every neuron in a specific layer. This collection of distributions constitutes the Visual DNA.
- The Comparison: We compute the vdna for the current environment and compare it to the pre-computed vdna of stored maps using the Earth Mover’s Distance (EMD). EMD provides a robust metric for how much one distribution must be transformed to match another.
- Ranking: The map with the vdna closest to the current live vdna is ranked as the best candidate for localisation.
This approach is "backbone-agnostic," meaning it functions effectively across various neural network architectures, including CosPlace and Mugs.
Experimental Setup
The system was evaluated on two distinct datasets representing different environmental challenges:
- Nordland Dataset: A 3,000km train journey in Norway recorded across four seasons. This dataset is ideal for testing "seasonal-semantic" changes (e.g., snow vs. foliage) without the interference of viewpoint changes, as the camera is fixed to a train on tracks.
- University Parks Dataset: Collected using a Clearpath Jackal UGV in Oxford, this dataset introduces real-world complexities, including varying lighting conditions (overcast vs. direct sun) and mild seasonal shifts in a wooded area.
Key Insights
Neuron Statistics as a Performance Proxy
A core finding of this research is that the way a neural network's neurons fire in response to a scene is a highly reliable indicator of localisation success. If the activation statistics of the live view match those of a stored experience, the localisation performance is likely to be high. This allows the system to predict performance based purely on visual similarity, making it functional in GPS-denied environments.
Efficiency and Safety
Despite the use of deep neural networks, the vdna comparison is computationally lightweight. Onboard a robot's CPU, the system can process 100 images and select an experience in under 16 seconds. Furthermore, the system exhibits "safe" failure modes; when it ranks a map incorrectly (e.g., selecting "Fall" instead of "Summer"), the actual localisation performance between those two maps is usually very similar. It rarely makes catastrophic errors, such as selecting a "Winter" map for a "Summer" deployment.
Results
Visual DNA significantly outperformed traditional baselines, such as Pixel Intensity and Fréchet Distance. On the Nordland dataset, vdna achieved a ranking error of only 0.23%, compared to 6.31% for pixel intensity and 11.76% for random selection. While simple brightness checks can sometimes identify day/night shifts, they fail during seasonal changes where semantic content shifts (e.g., a bright winter day vs. a bright summer day). Visual DNA handled these high-level semantic shifts with high accuracy, and the rankings produced correlated almost perfectly with actual Recall@1 metrics.
Conclusion and Future Work
This work demonstrates that dataset-to-dataset similarity, captured via deep neural activation statistics, is a powerful tool for experience selection in autonomous navigation. While the current system requires a "warmup" period of 45–100 frames to build a stable vdna profile, future work will explore single-frame localisation and "neuron pruning" to remove activations that are overly sensitive to appearance changes. By focusing on the most robust structural elements of a scene, we aim to move from using vdna solely for map selection to using it directly for the localisation process itself.
Summary updated