What You See Is What You Get: Experience Ranking with Deep Neural Dataset-to-Dataset Similarity for Topological Localisation

Matthew Gadd, Benjamin Ramtoula, Daniele De Martini, Paul Newman

TL;DR

Researchers from the University of Oxford have developed a way for robots to automatically select the best "visual memory" (map) for localization by comparing the statistical "fingerprint" of their current view to past experiences. Using a technique called Visual DNA, the system predicts which map will yield the highest accuracy—such as choosing a "Winter" map when it's snowing—without needing any ground-truth GPS or human intervention at runtime.

What Problem Does This Paper Address?

For an autonomous robot to know where it is, it often compares what it sees now to a stored map of the environment. However, the world changes: a park looks different in a summer sunset than on a snowy winter morning. This is known as domain shift.

If a robot tries to localize against every map it has ever recorded (Summer, Winter, Day, Night), it wastes massive amounts of computation. If it picks the wrong map, localization fails. The challenge is experience selection: how can a robot a priori (before actually trying to localize) rank its stored experiences and pick the one that best matches current conditions? Crucially, this must be done without access to "ground truth" (like GPS) to tell the robot if it's right or wrong.

Methods

The authors propose using Visual DNA (vdna) to measure the similarity between datasets.

  1. The Fingerprint: Instead of just looking at individual image features, they pass a sequence of images through a pre-trained deep neural network (like ResNet or a Vision Transformer). They record the distribution (histograms) of activations for every neuron in a specific layer. This collection of distributions is the "Visual DNA."
  2. The Comparison: When a robot starts moving, it collects a few "warmup frames." It computes the vdna for these frames and compares it to the pre-computed vdna of its stored maps using the Earth Mover’s Distance (EMD)—a metric that measures how different two distributions are.
  3. Ranking: The map with the vdna closest to the current "warmup" vdna is ranked as the best candidate for localization.

This approach is "backbone-agnostic," meaning it works with various neural network architectures (the authors tested CosPlace and Mugs).

Datasets

The study evaluated the method on two distinct types of data:

  • Nordland Dataset: A famous dataset of a 3,000km train journey in Norway recorded in four seasons. It is ideal for testing "seasonal-semantic" changes (snow vs. leaves) without the distraction of viewpoint changes, as the train is on fixed tracks.
  • University Parks Dataset: Data collected by the authors using a Clearpath Jackal UGV in Oxford. This dataset includes more complex real-world challenges, such as varying lighting conditions (overcast vs. direct sun) and mild seasonal shifts in a wooded area.

Key Insights and Ideas

  • Neuron Statistics as a Proxy for Performance: The core "aha" moment is that the way a neural network's neurons "fire" in response to a scene is a highly reliable indicator of how well that network will be able to localize in that scene. If the activation statistics match, the localization performance likely will too.
  • Divorce from Ground Truth: Most systems evaluate performance by checking against GPS. This system predicts performance based purely on visual similarity, making it functional in "GPS-denied" environments.
  • Efficiency on the Edge: Despite using large neural networks, the vdna comparison is lightweight. The authors demonstrated that a robot could process 100 images and select an experience in under 16 seconds using only an onboard CPU.
  • "Safe" Mistakes: The authors found that when the system did rank a map incorrectly (e.g., swapping "Fall" for "Summer"), the actual localization performance between those two maps was usually very similar. The system rarely makes "catastrophic" errors, like picking a "Winter" map for a "Summer" day.

Key Results

  • Superior Ranking: Visual DNA significantly outperformed baselines like "Pixel Intensity" (which just looks at brightness) and "Fréchet Distance."
  • Nordland Accuracy: On the Nordland dataset, vdna achieved a ranking error of only 0.23%, compared to 6.31% for pixel intensity and 11.76% for random selection.
  • Robustness to Seasonality: While simple brightness checks work for day/night shifts, they fail during seasonal changes (where a bright winter day looks nothing like a bright summer day). Visual DNA handled these high-level semantic shifts with ease.
  • Recall@1 Correlation: The rankings produced by vdna correlated almost perfectly with the actual Recall@1 (the standard metric for localization success) achieved during deployment.

Limitations and Future Work

  • The Warmup Requirement: The system currently requires a "warmup" period (e.g., 45–100 frames) to build a stable vdna profile. In a scenario where a robot needs to localize instantly from a single frame, this method might struggle.
  • Neuron Pruning: The authors suggest that future work could involve "pruning" neurons that are too sensitive to appearance changes, potentially creating even more robust "Visual DNA" that focuses only on the structural elements of a scene.
  • Sequential Localization: The authors aim to move from using vdna just for selecting a map to using it directly for the localization process itself.

Summary updated

Abstract

Recalling the most relevant visual memories for localisation or understanding a priori the likely outcome of localisation effort against a particular visual memory is useful for efficient and robust visual navigation. Solutions to this problem should be divorced from performance appraisal against ground truth – as this is not available at run-time – and should ideally be based on generalisable environmental observations. For this, we propose applying the recently developed Visual DNA as a highly scalable tool for comparing datasets of images – in this work, sequences of past (map) and live experiences. In the case of localisation, important dataset differences impacting performance are modes of appearance change, including weather, lighting, and season. Specifically, for any deep architecture which is used for place recognition by matching feature volumes at a particular layer, we use distribution measures to compare neuron-wise activation statistics between live images and multiple previously recorded past experiences, with a potentially large seasonal (winter/summer) or time of day (day/night) shift. We find that differences in these statistics correlate to performance when localising using a past experience with the same appearance gap. We validate our approach over the Nordland cross-season dataset as well as data from Oxford’s University Parks with lighting and mild seasonal change, showing excellent ability of our system to rank actual localisation performance across candidate experiences.
BibTeX
@inproceedings{gadd2023what,
  author = {Matthew Gadd and Benjamin Ramtoula and Daniele De Martini and Paul Newman},
  title = {{What You See Is What You Get: Experience Ranking with Deep Neural Dataset-to-Dataset Similarity for Topological Localisation}},
  booktitle = {ISER},
  year = {2023},
  url = {https://openreview.net/forum?id=CHqm23gLuU},
}

Loading related content...

Figure