TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing

Jiaming Wang, Diwen Liu, Jizhuo Chen, Harold SoharXiv 2025 · DOI · arXiv

Introduction

In robot navigation, topological maps represent the world as a graph where nodes signify "places" and edges represent "navigable paths." While significantly more efficient than dense 3D metric maps, topological mapping systems face a critical challenge: perceptual aliasing. This phenomenon occurs when two physically distinct locations appear identical to a robot’s sensors, such as two indistinguishable office corridors or repetitive outdoor terrain.

Examples of perceptual aliasing in indoor environments where distinct locations appear visually identical.

Until now, the field lacked a standardized evaluation protocol. Researchers often utilized disparate datasets and metrics, making it difficult to determine whether an algorithm's success was due to its robustness or simply an easier test environment. TOPO-Bench addresses this by providing a standardized framework and the first quantitative method for measuring dataset ambiguity.

Evaluation Methodology

TOPO-Bench introduces a principled evaluation protocol built upon three core pillars:

  1. Topological Consistency: The framework formalizes two properties: Edge Precision (ensuring nodes connected in the graph are physically close) and Edge Recall (ensuring physically close places are connected in the graph). These are measured using Geodesic and N-hop distances.
  2. Localization as a Surrogate: The authors provide a mathematical proof that localization accuracy—the ability to correctly identify the robot's current node—serves as a faithful surrogate for the overall structural consistency of the map.
  3. Quantifying Ambiguity: This novel contribution categorizes test cases based on visual similarity scores into three types: Positive Only (P.O.) (clear revisits), Ambiguous + Positive (A+P) (revisits with confusingly similar alternatives), and Ambiguous Only (A.O.) (new places that look like known ones).
Comparison of Geodesic and N-hop distances used to define topological consistency and evaluate mapping accuracy.

To synthesize these factors, the authors introduce Balanced Localization Accuracy (BLA). This metric uses the geometric mean of accuracies across the three ambiguity categories, preventing systems from appearing successful by excelling in only one area while failing in others.

Benchmarking and Insights

The authors curated a massive benchmark from six high-quality datasets (OpenLORIS, Habitat-Sim, Rawseeds, Oxford RobotCar, ROVER, and RELLIS-3D), covering 25 diverse indoor, urban, and off-road environments.

Key findings from benchmarking state-of-the-art baselines include:

  • The Safety vs. Recall Trade-off: Current systems often maintain "safety" (avoiding false loops) only by being extremely conservative. When thresholds are set to avoid errors in "Ambiguous Only" cases, the ability to recognize actual revisits (P.O. and A+P) typically collapses.
  • Deep Learning Superiority: Baselines utilizing modern deep-learned Visual Place Recognition (VPR) models, such as MegaLoc, significantly outperformed classical methods like FAB-MAP and RatSLAM.
  • Sequence Matching Limitations: While matching sequences of images is a common strategy to combat aliasing, it is highly parameter-sensitive. Excessive sequence lengths can break matches due to viewpoint changes, while short sequences may collapse into simple greedy matching.

Conclusion

Benchmarking reveals that no current method is truly robust, particularly in A+P scenarios where revisits are ambiguous. The Probabilistic Belief Update (PBU) method achieved the highest BLA (0.295), yet this low absolute score underscores the significant room for improvement in the field. Future work must focus on more sophisticated belief models and the integration of non-visual or semantic cues to effectively disambiguate identical-looking environments. TOPO-Bench provides the necessary tools to measure this progress quantitatively.

Summary updated

BibTeX
@misc{wang2025topo,
  author = {Jiaming Wang and Diwen Liu and Jizhuo Chen and Harold Soh},
  title = {{TOPO-Bench: An Open-Source Topological Mapping Evaluation Framework with Quantifiable Perceptual Aliasing}},
  year = {2025},
  eprint = {2510.04100},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.04100},
}
Figure