Multimodal Fusion SLAM With Fourier Attention

Youjie Zhou, Guofeng Mei, Yiming Wang, Yi Wan, Fabio Poiesi

This paper, recently accepted for IEEE RAL, introduces FMF-SLAM, a robust multimodal SLAM system designed to navigate where traditional visual sensors fail. By combining RGB and depth data through a novel, computationally efficient Fourier-based attention mechanism, the authors demonstrate state-of-the-art localization even in total darkness or high-noise environments.

TL;DR

FMF-SLAM is a learning-based multimodal SLAM system that uses Fast Fourier Transforms (FFT) to perform efficient feature fusion between RGB and depth signals. It outperforms existing methods in visually challenging conditions—like darkness and rapid lighting changes—while maintaining the efficiency required for real-time deployment on robotic platforms.

What Problem Does This Paper Address?

Visual Simultaneous Localization and Mapping (SLAM) is the "eyes" of an autonomous robot. However, standard visual SLAM often goes "blind" in environments with low light, glare, or heavy sensor noise. While researchers have turned to deep learning and multimodal fusion (combining RGB with Depth) to solve this, two major hurdles remain:

  1. Computational Cost: Standard "Attention" mechanisms (the backbone of modern AI) use dot-product operations that are computationally expensive, making them hard to run in real-time on mobile robots.
  2. Feature Disparity: RGB and depth sensors produce data with different magnitudes and characteristics. Simply "stacking" them often leads to poor integration where one modality ignores the other.

Methods

FMF-SLAM builds upon the architecture of DROID-SLAM but introduces three significant innovations:

  • Fourier-Based Attention: Instead of traditional spatial-domain attention, the authors use the Fast Fourier Transform (FFT) to move features into the frequency domain. In this domain, complex "attention" correlations can be calculated using simple element-wise products. This drastically reduces the mathematical overhead while still capturing the global context needed for accurate tracking.
  • Multi-Scale Knowledge Distillation: To ensure the RGB and depth branches work in harmony, the system uses a distillation technique. It forces the two branches to align their internal representations across three levels: overall magnitude ($L_{L2}$), spatial similarity ($L_s$), and channel-wise differences ($L_c$). This ensures that even if the RGB signal is weak (e.g., in the dark), the features extracted are still consistent with the depth data.
  • Robot Integration: The system isn't just a theoretical model; it integrates a global positioning module (GNSS-RTK) and an Extended Kalman Filter (EKF) to fuse visual data with global coordinates, followed by Global Bundle Adjustment (BA) to minimize trajectory drift.

Datasets

The authors evaluated FMF-SLAM across a mix of simulated and real-world data:

  • TUM & TartanAir: Standard benchmarks for visual SLAM. They specifically focused on the "challenging" subsets of TartanAir (fast motion, dimmed light, and night scenes).
  • Realtest-GT & Realtest-Visual: A custom dataset recorded using a RealSense L515 camera. Crucially, they used a high-precision laser tracker to provide millimeter-accurate ground truth for their indoor robotic tests.

Key Insights and Ideas

  • The "Fourier Trick" for Efficiency: The most elegant insight is using the Convolution Theorem—the idea that convolution in the spatial domain is multiplication in the frequency domain. By applying this to attention mechanisms, they achieve the "global reach" of Transformers without the $O(N^2)$ computational cost.
  • Cross-Modal Synergy: The ablation studies show that adding a second branch (Depth) helps, but adding the Fourier Attention and Distillation is what actually stabilizes the system. Without these, the system still struggles with "outliers" (tracking failures) in extreme darkness.
  • Zero-Outlier Performance: In tests where classical methods (ORB-SLAM3) and even recent learning-based methods (DROID-SLAM) suffered from tracking failures, FMF-SLAM maintained a 100% success rate across all evaluated sequences.

Key Results

  • Accuracy: On the TartanAir dataset, FMF-SLAM achieved an Average Absolute Trajectory Error (ATE) of 1.07 cm, significantly beating DROID-SLAM (6.02 cm) and RESLAM (14.34 cm).
  • Robustness: In "Dark" scenarios, FMF-SLAM reduced the error by over 50% compared to its closest competitors.
  • Speed: The system runs at approximately 77.76 ms per frame on an Nvidia 3090, making it viable for real-time robotics.
  • Real-World Success: The system was successfully deployed on a security robot, demonstrating an accumulation error of only 0.06% over a 147-meter outdoor path.

Limitations and Future Work

While FMF-SLAM is highly robust, the authors note that obtaining stable depth information in large-scale outdoor environments remains a challenge for sensors like the RealSense L515. To compensate, they utilized ZoeDepth (a pre-trained depth estimation model) for their outdoor robot trials.

Future work could involve tighter integration of LiDAR data or exploring how these Fourier-based encoders handle even more diverse modalities, such as thermal imaging, for search-and-rescue operations in smoke-filled environments.

Summary updated

Abstract

Visual SLAM is particularly challenging in environments affected by noise, varying lighting conditions, and darkness. Learning-based optical flow algorithms can leverage multiple modalities to address these challenges, but traditional optical flow-based visual SLAM approaches often require significant computational resources. To overcome this limitation, we propose FMF-SLAM, an efficient multimodal fusion SLAM method that utilizes fast Fourier transform (FFT) to enhance the algorithm efficiency. Specifically, we introduce a novel Fourier-based self-attention and cross-attention mechanism to extract features from RGB and depth signals. We further enhance the interaction of multimodal features by incorporating multi-scale knowledge distillation across modalities. We also demonstrate the practical feasibility of FMF-SLAM in real-world scenarios with real time performance by integrating it with a security robot by fusing with a global positioning module GNSS-RTK and global Bundle Adjustment. Our approach is validated using video sequences from TUM, TartanAir, and our real-world datasets, showcasing state-of-the-art performance under noisy, varying lighting, and dark conditions.
BibTeX
@article{zhou2025multimodal,
  author = {Youjie Zhou and Guofeng Mei and Yiming Wang and Yi Wan and Fabio Poiesi},
  title = {{Multimodal Fusion SLAM With Fourier Attention}},
  journal = {IEEE Robotics Autom. Lett.},
  year = {2025},
  url = {https://openreview.net/forum?id=tNxtEbpVtT},
}

Loading related content...

Figure