Multi-Modal Recurrent Fusion for Indoor Localization

Jianyuan Yu, Pu Wang, Toshiaki Koike-Akino, Philip V. Orlik

Indoor localization is the "GPS-less" frontier of navigation. While we’ve mastered positioning under the open sky, doing the same inside a complex building—where walls block signals and interference is rampant—remains a significant challenge. A new paper from Mitsubishi Electric Research Laboratories (MERL) proposes a clever way to solve this by blending multiple wireless signals using a "recurrent fusion" neural network that learns which sensors to trust in real-time.

TL;DR

The researchers developed a multi-stream recurrent neural network (LSTM) that fuses Wi-Fi, Ultra-Wideband (UWB), and Inertial Measurement Unit (IMU) data for indoor positioning. By learning "importance weights" from the immediate past states of each sensor, the system can dynamically ignore noisy or blocked signals, achieving a mean localization error as low as 6 centimeters.

What Problem Does This Paper Address?

Indoor localization usually relies on one of three things: Wi-Fi (ubiquitous but noisy), UWB (precise but fails if the line-of-sight is blocked), or IMUs (good for short-term tracking but prone to "drift" over time).

The core challenge is modality uncertainty. For example, if a robot moves behind a thick concrete pillar, the UWB signal might become garbage while the IMU remains reliable. Traditional fusion methods (like Kalman Filters) often require complex mathematical models of how sensors fail. This paper asks: Can we train a neural network to figure out sensor reliability on its own, just by looking at the recent history of the data?

Methods

The authors propose a Multi-Stream Recurrent Fusion Network. Here’s the breakdown:

  1. Multi-Stream LSTMs: Each sensor type (Wi-Fi RSSI, Wi-Fi CSI, UWB, and IMU) gets its own dedicated Long Short-Term Memory (LSTM) pipeline. This allows the model to learn the specific temporal "evolution" of each data type.
  2. Data Curation: Raw Wi-Fi Channel State Information (CSI) is messy. The authors use polynomial fitting to compress this complex data into a manageable set of coefficients, reducing overhead while keeping the "shape" of the signal.
  3. Recurrent Fusion Block (The "Secret Sauce"): Instead of just concatenating all sensor data, the model looks at the immediate past hidden states of each stream. It passes these through a fully connected layer to calculate a "quality measure" or importance weight ($\alpha$) for each sensor.
  4. Softmax Normalization: These weights are normalized so they sum to one. If the UWB stream's past states suggest it’s currently unreliable (e.g., due to a sudden drop in signal quality), its weight decreases, and the model relies more on the other sensors to predict the $(x, y)$ coordinates.

Datasets

The model was evaluated using the SPAWC2021 multi-modal localization dataset.

  • Scale: ~750,000 samples collected by a robot traversing nearly 2 kilometers in a multi-room environment.
  • Sensors: 11 Wi-Fi anchors (RSSI and 64-subcarrier CSI), 3 UWB anchors (range and power), and a 9-axis IMU.
  • Ground Truth: High-precision coordinates obtained via Lidar-based SLAM.

Key Insights and Ideas

  • Past as Prologue: The most important insight is that a sensor's immediate history is a fantastic predictor of its current accuracy. By looking at just 1 or 2 past hidden states ($F=1$), the model can effectively "gate" the fusion process.
  • CSI vs. RSSI: While many systems use simple signal strength (RSSI), this model leverages the fine-grained Channel State Information (CSI), which provides a much richer "fingerprint" of the indoor environment.
  • Simplicity Wins: Interestingly, using a shorter sequence length ($T=10$) performed better than a longer one ($T=30$). This suggests that for high-speed indoor localization, the most recent data is far more relevant than long-term history.

Key Results

The proposed Recurrent Fusion (RF) method significantly outperformed traditional trilateration and standard convolutional neural networks (CNNs).

  • Accuracy: The best configuration (Bi-directional LSTM, $T=10$, $F=1$) achieved a mean error of 0.06 meters (6 cm) and a median error of 0.05 meters.
  • Comparison: For context, the best non-recurrent fusion method (RIU-Conv) had a mean error of 0.26 meters. The RF method improved accuracy by nearly 4x.
  • Robustness: Even in "dead zones" where UWB coverage was zero (due to non-line-of-sight conditions), the fusion model maintained high accuracy by dynamically shifting weight to Wi-Fi and IMU data.

Limitations and Future Work

  • Redundancy: The authors noted that increasing the number of past hidden states used for weight calculation ($F=2$) actually decreased performance, likely due to overfitting or redundant information.
  • Computational Load: While LSTMs are powerful, running four parallel streams plus a fusion block might be heavy for low-power IoT devices. Future work could explore lighter architectures like GRUs or 1D-CNNs for the individual streams.
  • Static Anchors: The system relies on fixed Wi-Fi and UWB anchor locations. Adapting this to ad-hoc environments where anchor positions aren't perfectly known remains a challenge.

Summary updated

Abstract

This paper considers indoor localization using multi-modal wireless signals including Wi-Fi, inertial measurement unit (IMU), and ultra-wideband (UWB). By formulating the localization as a multi-modal sequence regression problem, a multi-stream recurrent fusion method is proposed to combine the current hidden state of each modality in the context of recurrent neural networks while accounting for the modality uncertainty which is directly learned from its own immediate past states. The proposed method was evaluated on the large-scale SPAWC2021 multi-modal localization dataset and compared with a wide range of baseline methods including the trilateration method, traditional fingerprinting methods, and convolution network-based methods.
BibTeX
@inproceedings{yu2022multi,
  author = {Jianyuan Yu and Pu Wang and Toshiaki Koike-Akino and Philip V. Orlik},
  title = {{Multi-Modal Recurrent Fusion for Indoor Localization}},
  booktitle = {IEEE International Conference on Acoustics, Speech and Signal Processing},
  year = {2022},
  url = {https://openreview.net/forum?id=DkIMmq9iynQ},
}

Loading related content...

Figure