Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned From the IPIN 2018 Competition

Valérie Renaudin, Miguel Ortiz, Johan Perul, Joaquín Torres-Sospedra, Antonio Ramón Jiménez, Antoni Pérez-Navarro, Germán Martín Mendoza-Silva, Fernando Seco, Yael Landau, Revital Marbel, Boaz Ben-Moshe, Xingyu Zheng, Feng Ye, Jian Kuang, Yu Li, Xiaoji Niu, Vlad Landa, Shlomi Hacohen, Nir Shvalb, Chuanhua Lu, Hideaki Uchiyama, Diego Thomas, Atsushi Shimada, Rin-Ichiro Taniguchi, Zhenxing Ding, Feng Xu, Nikolai Kronenwett, Blagovest Vladimirov, Soyeon Lee, Eunyoung Cho, Sungwoo Jun, Chang-Eun Lee, Sangjoon Park, Yonghyun Lee, Jehyeok Rew, Changjun Park, Hyeongyo Jeong, Jaeseung Han, Keumryeol Lee, Wenchao Zhang, Xianghong Li, Dongyan Wei, Ying Zhang, So Young Park, Stefan Knauth, Georgios Pipelidis, Nikolaos Tsiamitros, Tomás Lungenstrass, Juan Pablo Morales, Jens Trogh, David Plets, Miroslav Opiela, Shih-Hau Fang, Yu Tsao, Ying-Ren Chien, Shi-Shen Yang, Shih-Jyun Ye, Muhammad Usman Ali, Soojung Hur, Yongwan ParkIEEE Access 2019 · DOI

Indoor positioning remains the "final frontier" of navigation. While GPS has revolutionized outdoor transit, its signals fail to penetrate building structures, leaving a significant gap in our ability to navigate complex indoor spaces. The IPIN 2018 Competition sought to address this by moving beyond controlled laboratory settings into the Atlantis Shopping Mall in Nantes, France—a massive, 14,000 m² multi-floor environment characterized by dense crowds, metallic structures, and significant signal interference.

The Challenge of Infrastructure-Free Positioning

The primary objective of the competition was to evaluate "infrastructure-free" systems: technologies capable of locating a person using only wearable sensors or smartphones, without the aid of dedicated beacons or specialized hardware. A major hurdle in such research is establishing a reliable "ground truth." To solve this, organizers conducted a comprehensive topographical survey using theodolites, differential GNSS, and 3D scanners to map 180 waypoints with a precision of 10 cm.

Methodology and Competition Tracks

Participants evaluating positioning systems within the complex environment of the Atlantis Shopping Mall.

The competition was organized into four distinct tracks to evaluate different technological approaches:

  • Track 1 (On-site, Camera-based): Real-time positioning utilizing computer vision and Augmented Reality (AR) frameworks.
  • Track 2 (On-site, Non-camera): Real-time positioning using inertial sensors, barometers, and existing Wi-Fi infrastructure.
  • Track 3 (Off-site, Smartphone): Post-processing of raw smartphone data, including Wi-Fi RSS, magnetic fields, and IMU logs.
  • Track 4 (Off-site, Foot-mounted): Post-processing of data from high-end foot-mounted Inertial Measurement Units (IMU).

Performance was measured using the 75th percentile error. A critical component of the scoring was a 15-meter penalty for incorrect floor detection, reflecting the catastrophic nature of floor-level errors in real-world navigation.

Key Insights and Results

The results revealed a significant performance gap between "off-site" and "on-site" systems. Off-site teams, who had 16 weeks to refine their algorithms, achieved sub-meter accuracy (0.90m). In contrast, on-site teams—who had less than 24 hours for calibration—faced much steeper challenges.

Track Best Score (75th Percentile Error)
Track 1 (Camera, On-site) 11.70 m
Track 2 (Non-camera, On-site) 5.50 m
Track 3 (Smartphone, Off-site) 0.90 m
Track 4 (Foot-mounted, Off-site) 1.30 m

Critical Findings:

  • The Calibration Gap: The disparity between off-site and on-site results highlights the difficulty of creating "plug-and-play" systems for real-world environments.
  • Magnetic Fingerprinting: Top performers in the smartphone track successfully leveraged magnetic field anomalies as unique identifiers to supplement Wi-Fi data.
  • Environmental Obstacles: In the on-site tracks, non-camera systems actually outperformed camera-based systems. This was largely due to the mall's dynamic environment, where specular reflections and moving crowds frequently disrupted visual odometry and SLAM algorithms.
  • Floor Detection: The 15m penalty ensured that robust vertical positioning was as important as horizontal accuracy.

Future Directions

The IPIN 2018 competition underscored that while indoor positioning algorithms are maturing, environmental dynamics remain a significant barrier. Future research must focus on filtering out dynamic human obstacles and developing systems that require zero prior site knowledge to function effectively. The datasets generated during this competition, including over 1 km of walking paths and geo-referenced waypoints, remain available as a benchmark for the global research community.

Summary updated

BibTeX
@article{renaudin2019evaluating,
  author = {Valérie Renaudin and Miguel Ortiz and Johan Perul and Joaquín Torres-Sospedra and Antonio Ramón Jiménez and Antoni Pérez-Navarro and Germán Martín Mendoza-Silva and Fernando Seco and Yael Landau and Revital Marbel and Boaz Ben-Moshe and Xingyu Zheng and Feng Ye and Jian Kuang and Yu Li and Xiaoji Niu and Vlad Landa and Shlomi Hacohen and Nir Shvalb and Chuanhua Lu and Hideaki Uchiyama and Diego Thomas and Atsushi Shimada and Rin-Ichiro Taniguchi and Zhenxing Ding and Feng Xu and Nikolai Kronenwett and Blagovest Vladimirov and Soyeon Lee and Eunyoung Cho and Sungwoo Jun and Chang-Eun Lee and Sangjoon Park and Yonghyun Lee and Jehyeok Rew and Changjun Park and Hyeongyo Jeong and Jaeseung Han and Keumryeol Lee and Wenchao Zhang and Xianghong Li and Dongyan Wei and Ying Zhang and So Young Park and Chan Gook Park and Stefan Knauth and Georgios Pipelidis and Nikolaos Tsiamitros and Tomás Lungenstrass and Juan Pablo Morales and Jens Trogh and David Plets and Miroslav Opiela and Shih-Hau Fang and Yu Tsao and Ying-Ren Chien and Shi-Shen Yang and Shih-Jyun Ye and Muhammad Usman Ali and Soojung Hur and Yongwan Park},
  title = {{Evaluating Indoor Positioning Systems in a Shopping Mall: The Lessons Learned From the IPIN 2018 Competition}},
  journal = {IEEE Access},
  year = {2019},
  doi = {10.1109/access.2019.2944389},
}
Figure