Off-Line Evaluation of Mobile-Centric Indoor Positioning Systems: The Experiences from the 2017 IPIN Competition

Joaquín Torres-Sospedra, Antonio R. Jiménez, Adriano Moreira, Tomás Lungenstrass, Wei-Chung Lu, Stefan Knauth, Germán Martín Mendoza-Silva, Fernando Seco, Antoni Pérez-Navarro, Maria João Nicolau, António Costa, Filipe Meneses, Joaquín Farina, J. C. Morales, Wen-Chen Lu, Ho-Ti Cheng, Shi-Shen Yang, Shih‐Hau Fang, Ying‐Ren Chien, Yu TsaoSensors 2018 · DOI

TL;DR

This paper details the 2017 IPIN competition's smartphone-based indoor localization track, highlighting the challenges of fair evaluation in this rapidly evolving field. By analyzing the competition's results, datasets, and lessons learned, the authors advocate for standardized evaluation frameworks and the creation of robust benchmark strategies to advance indoor positioning research.

What Problem Does This Paper Address?

The burgeoning market for indoor positioning systems, driven by smartphone ubiquity, faces a significant hurdle: the lack of standardized evaluation methods. Researchers often test their systems in proprietary environments using unique metrics, making direct comparisons between different solutions nearly impossible. This "evaluation diversity problem" hinders progress by preventing the reproduction of results and the objective assessment of system quality across various scenarios. The paper addresses this by presenting the 2017 IPIN competition as a concrete step towards establishing fair, reproducible, and comprehensive evaluation frameworks for mobile-centric indoor positioning.

Methods

The core of the paper revolves around the "Smartphone-based (off-site) track" of the 2017 IPIN competition. This track employed a standardized approach where organizers provided pre-collected datasets for training, validation, and testing. This ensured all competitors used identical data, eliminating bias from custom infrastructure deployment or calibration. The evaluation metric used was the third quartile of the positioning error, with penalties for incorrect floor or building identification. The paper also details the approaches of the competing teams, which primarily involved Wi-Fi fingerprinting, often augmented with sensor fusion (IMU, BLE, etc.) and advanced algorithms like k-nearest neighbors and particle filters.

Datasets

The competition utilized datasets collected from three buildings (CAR, UJITI, and UJIUB) using a dedicated Android application (GetSensorData v2). These datasets included sensor measurements (Wi-Fi, BLE, accelerometer, magnetometer, gyroscope, compass) and ground truth location data (POSI). The data was provided in logfile format, with separate training, validation, and test sets. The paper notes that these datasets were made publicly available, building upon data from the previous year's competition, fostering reproducibility and further research.

Key Insights and Ideas

  • The critical need for standardized evaluation: The paper strongly emphasizes that without common benchmarks and datasets, progress in indoor positioning is significantly hampered. Competitions like IPIN are vital for generating this standardized data.
  • Diversity in datasets is key: The analysis reveals that no single system performed best across all scenarios. This highlights the importance of evaluating systems on diverse datasets representing various environments, building layouts, and even actor movements to truly assess robustness.
  • Competition drives innovation: The structured environment of the IPIN competition incentivizes teams to develop highly competitive and accurate systems, pushing beyond mere prototypes.
  • Academia and industry collaboration: Competitions serve as valuable platforms for both academic research and industrial development to showcase their advancements and foster synergies.
  • Off-site vs. On-site evaluation trade-offs: While off-site evaluations (like IPIN Track 3) offer more comprehensive data collection and fairness, they miss the real-time deployment and adaptation challenges faced in on-site competitions.

Key Results

The 2017 IPIN competition saw the UMinho team emerge as the winner with a third quartile positioning error of 3.48 meters. The AraraDS team closely followed with 3.53 meters, demonstrating a high level of performance from both. The Yai and HFTS teams secured third and fourth places, respectively. Detailed analysis showed that while UMinho had the best overall score, other teams excelled in specific logfiles, underscoring the context-dependent nature of indoor positioning performance. The competition provided a rich dataset for evaluating systems across multiple buildings and trajectories, with a total of 505 evaluation points.

Limitations and Future Work

The authors acknowledge several limitations and suggest future work:

  • Minimizing unexpected events: Competitors suggested minimizing "unexpected events" like missing stairs in training data to avoid unfair penalties.
  • Floor transition coding: A more explicit coding for floor transitions (e.g., a special floor code) could improve fairness.
  • Metric fairness: The penalties for floor/building misidentification were considered high by some competitors. Future editions might explore more realistic distance metrics that account for walkable distance.
  • Data registration methods: While the competition used continuous data logging, some fingerprinting-focused teams prefer static or very slow motion data for more stable Wi-Fi fingerprints. Future competitions could consider incorporating sites with dense static data.
  • Real-time component: Off-site competitions, while comprehensive, do not evaluate the real-time adaptation and deployment capabilities of systems.
  • Standardization: The paper advocates for continued efforts towards standardizing evaluation procedures and creating widely adopted benchmark strategies.

Summary updated

Abstract

The development of indoor positioning solutions using smartphones is a growing activity with an enormous potential for everyday life and professional applications. The research activities on this topic concentrate on the development of new positioning solutions that are tested in specific environments under their own evaluation metrics. To explore the real positioning quality of smartphone-based solutions and their capabilities for seamlessly adapting to different scenarios, it is needed to find fair evaluation frameworks. The design of competitions using extensive pre-recorded datasets is a valid way to generate open data for comparing the different solutions created by research teams. In this paper, we discuss the details of the 2017 IPIN indoor localization competition, the different datasets created, the teams participating in the event, and the results they obtained. We compare these results with other competition-based approaches (Microsoft and Perf-loc) and on-line evaluation web sites. The lessons learned by organising these competitions and the benefits for the community are addressed along the paper. Our analysis paves the way for future developments on the standardization of evaluations and for creating a widely-adopted benchmark strategy for researchers and companies in the field.
BibTeX
@inproceedings{torressospedra2018off,
  author = {Joaquín Torres-Sospedra and Antonio R. Jiménez and Adriano Moreira and Tomás Lungenstrass and Wei-Chung Lu and Stefan Knauth and Germán Martín Mendoza-Silva and Fernando Seco and Antoni Pérez-Navarro and Maria João Nicolau and António Costa and Filipe Meneses and Joaquín Farina and J. C. Morales and Wen-Chen Lu and Ho-Ti Cheng and Shi-Shen Yang and Shih‐Hau Fang and Ying‐Ren Chien and Yu Tsao},
  title = {{Off-Line Evaluation of Mobile-Centric Indoor Positioning Systems: The Experiences from the 2017 IPIN Competition}},
  booktitle = {Sensors},
  year = {2018},
  doi = {10.3390/s18020487},
}
Figure