Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization

Grigorios G. Anagnostopoulos, Alexandros KalousisSensors 2022 · OpenReview

TL;DR

This paper addresses the critical lack of reproducible benchmarking for Dynamic Accuracy Estimation (DAE) methods in localization systems. The authors introduce an open-source framework, evaluate various DAE methods on diverse datasets, and provide valuable insights into their performance, aiming to guide future research and practical deployment.

What Problem Does This Paper Address?

Indoor Positioning Systems (IPS) and outdoor IoT systems using Low-Power Wide-Area Networks (LPWANs) often provide users with a location estimate. However, a crucial piece of information often missing or inconsistently reported is the confidence in that estimate, known as Dynamic Accuracy Estimation (DAE). This DAE is vital for users to understand the reliability of their position and for applications to make informed decisions. The problem is that the field suffers from a lack of standardized comparisons: new methods rarely build on previous ones, evaluation metrics are inconsistent, open data is scarce, and code is rarely shared. This makes it difficult for practitioners to choose the best DAE method for their specific needs.

Methods

The authors tackle this problem by developing a reproducible benchmarking framework. This framework involves:

  • Comprehensive Literature Review: They systematically review and categorize existing DAE methods into rule-based and data-driven approaches, establishing a consistent terminology.
  • Implementation of Existing Methods: They implement several key DAE methods from the literature, including popular rule-based approaches like "Best Candidate Set" (Lemelson et al.) and "DAE Marcus," as well as data-driven methods.
  • Development of Naive Baselines: To provide a lower bound for performance, they introduce four naive baselines: constant mean error, constant median error, uniform random error, and normal random error.
  • Evaluation Metrics: They utilize a rich set of evaluation metrics, including absolute error statistics (mean, median, percentiles, standard deviation), overestimation/underestimation metrics, and correlation coefficients (Pearson and Spearman).
  • Open-Source Framework: Crucially, they make their code and the processed datasets publicly available, enabling reproducibility and future research.

Datasets

The study utilizes a combination of indoor and outdoor datasets to ensure a diverse evaluation:

  • Outdoor Datasets:
    • Sigfox: An outdoor dataset of Received Signal Strength (RSS) values collected in Antwerp, Belgium, using Sigfox technology.
  • LoRaWAN v1.3 reduced: An outdoor dataset using LoRaWAN technology, also from Antwerp, with a focus on messages received from at least three gateways.
  • Indoor Datasets:
    • DSI: An indoor Wi-Fi dataset collected at the University of Minho, Portugal, including both "radio map" (grid-based) and "trajectory" (pedestrian path) samples. A reduced version of the "radio map" was used.
  • MAN: An indoor Wi-Fi dataset collected at the University of Mannheim, Germany, focusing on a single floor. A reduced version with one sample per location was used.

The authors emphasize the importance of using publicly available datasets and have made their processed versions available.

Key Insights and Ideas

  • Data-Driven Methods Shine (Mostly): In outdoor environments (Sigfox, LoRaWAN) and one indoor dataset (MAN), data-driven methods, particularly those incorporating location estimates (DAEDDL), generally outperform rule-based methods in terms of absolute error.
  • Indoor Anomaly: The DSI indoor dataset presents an exception, where rule-based methods (DAEzou, DAEMarcus) outperform data-driven ones. This is hypothesized to be due to the smaller training sample size for the DAE model in this specific dataset.
  • Correlation vs. Accuracy: Rule-based methods (DAEzou, DAEMarcus) show strong performance in ranking location estimates by accuracy (Spearman's rank correlation), even if they are less precise in estimating the exact error value compared to data-driven methods. This suggests potential for improvement through scaling or calibration.
  • Overestimation vs. Underestimation Matters: Different use cases might prioritize different types of errors. Rule-based methods tend to overestimate more, while data-driven methods can have higher underestimation errors. This distinction is crucial for practical deployment.
  • Reproducibility is Key: The paper strongly advocates for open code and standardized datasets to foster consistent comparisons and advance the field.

Key Results

  • Data-driven methods (DAEDD, DAEDDL) generally achieved lower absolute DAE errors across outdoor datasets and the MAN indoor dataset. For instance, on the LoRaWAN dataset, DAEDDL achieved a mean error of 138m compared to 223m for DAELemelson.
  • Rule-based methods (DAEzou, DAEMarcus) showed competitive performance in ranking estimates (Spearman correlation), often matching or slightly exceeding data-driven methods in this aspect.
  • DAELemelson consistently performed the worst across most metrics and datasets, often performing worse than naive baselines.
  • The DSI indoor dataset showed a reversal, with rule-based methods outperforming data-driven ones in absolute error metrics.
  • Correlation coefficients (Pearson) were generally higher for data-driven methods in datasets where they excelled, indicating a better ability to estimate the magnitude of the error.

Limitations and Future Work

  • Dataset Size Impact: The authors hypothesize that the smaller training data size for the DSI dataset might explain the atypical performance of data-driven methods there.
  • Rule-Based Method Improvement: The paper suggests that rule-based methods could be improved by introducing scaling factors or nonlinear transformations to better estimate error values, potentially closing the gap with data-driven methods.
  • Sequence-Based Methods: The current work focuses on DAE based on individual signal receptions. Future work could explore comparisons with sequence-based methods (e.g., Elbakly and Youssef [11], Khandker et al. [22]).
  • Theoretical Approaches: The authors mention a recent theoretical approach for error approximation (Perez-Navarro [44]) that warrants further investigation.
  • Broader Method Exploration: The paper aims to facilitate the exploration of new DAE methods within their benchmarking framework.

Summary updated

Abstract

Despite the great attention that the research community has paid to the creation of novel indoor positioning methods, a rather limited volume of works has focused on the confidence that Indoor Positioning Systems (IPS) assign to the position estimates that they produce. The concept of estimating, dynamically, the accuracy of the position estimates provided by an IPS has been sporadically studied in the literature of the field. Recently, this concept has started being studied as well in the context of outdoor positioning systems of Internet of Things (IoT) based on Low-Power Wide-Area Networks (LPWANs). What is problematic is that the consistent comparison of the proposed methods is quasi nonexistent: new methods rarely use previous ones as baselines; often, a small number of evaluation metrics are reported while different metrics are reported among different relevant publications, the use of open data is rare, and the publication of open code is absent. In this work, we present an open-source, reproducible benchmarking framework for evaluating and consistently comparing various methods of Dynamic Accuracy Estimation (DAE). This work reviews the relevant literature, presenting in a consistent terminology commonalities and differences and discussing baselines and evaluation metrics. Moreover, it evaluates multiple methods of DAE using open data, open code, and a rich set of relevant evaluation metrics. This is the first work aiming to establish the state of the art of methods of DAE determination in IPS and in LPWAN positioning systems, through an open, transparent, holistic, reproducible, and consistent evaluation of the methods proposed in the relevant literature.
BibTeX
@inproceedings{anagnostopoulos2022can,
  author = {Grigorios G. Anagnostopoulos and Alexandros Kalousis},
  title = {{Can I Trust This Location Estimate? Reproducibly Benchmarking the Methods of Dynamic Accuracy Estimation of Localization}},
  booktitle = {Sensors},
  year = {2022},
  url = {https://openreview.net/forum?id=iFri6MpmgV},
}

Loading related content...

Figure