Abstract
Visual SLAM is particularly challenging in environments affected by noise, varying lighting conditions, and darkness. Learning-based optical flow algorithms can leverage multiple modalities to address these challenges, but traditional optical flow-based visual SLAM approaches often require significant computational resources. To overcome this limitation, we propose FMF-SLAM, an efficient multimodal fusion SLAM method that utilizes fast Fourier transform (FFT) to enhance the algorithm efficiency. Specifically, we introduce a novel Fourier-based self-attention and cross-attention mechanism to extract features from RGB and depth signals. We further enhance the interaction of multimodal features by incorporating multi-scale knowledge distillation across modalities. We also demonstrate the practical feasibility of FMF-SLAM in real-world scenarios with real-time performance by integrating it with a security robot by fusing with a global positioning module GNSS-RTK and global Bundle Adjustment. Our approach is validated using video sequences from TUM, TartanAir, and our real-world datasets, showcasing state-of-the-art performance under noisy, varying lighting, and dark conditions.
Introduction
Simultaneous Localization and Mapping (SLAM) serves as the "eyes" of autonomous systems. However, standard visual SLAM systems frequently struggle in adverse conditions such as total darkness, glare, or heavy sensor noise. While deep learning and multimodal fusion (combining RGB with depth data) have emerged as promising solutions, they face two primary hurdles: the high computational cost of standard attention mechanisms and the difficulty of effectively integrating disparate data types.
In this work, we present FMF-SLAM, a robust multimodal system designed to maintain high-precision localization where traditional sensors fail. By introducing a novel Fourier-based attention mechanism and a multi-scale knowledge distillation framework, FMF-SLAM achieves state-of-the-art performance with the efficiency required for real-time robotic deployment.
Methodology
Fourier-Based Attention Mechanism
The core innovation of FMF-SLAM is the transition from spatial-domain attention to the frequency domain. Standard "Attention" mechanisms, while powerful, rely on dot-product operations that scale quadratically with input size, making them computationally prohibitive for real-time mobile robotics.
FMF-SLAM utilizes the Fast Fourier Transform (FFT) to move features into the frequency domain. According to the Convolution Theorem, complex correlations in the spatial domain can be calculated as simple element-wise products in the frequency domain. This "Fourier Trick" allows the system to capture global context and perform cross-modal fusion between RGB and depth signals with significantly reduced mathematical overhead.
Multi-Scale Knowledge Distillation
To ensure that the RGB and depth branches work in harmony, we implement a multi-scale knowledge distillation technique. RGB and depth sensors produce data with vastly different characteristics; simply concatenating these signals often leads to suboptimal integration.
Our distillation framework forces the two branches to align their internal representations across three distinct levels:
- Magnitude Alignment ($L_{L2}$): Ensuring the scale of features is consistent.
- Spatial Similarity ($L_s$): Aligning the structural information captured by both sensors.
- Channel-wise Differences ($L_c$): Refining the feature selection process.
This ensures that even when the RGB signal is severely degraded (e.g., in total darkness), the features extracted are stabilized by the depth data.
Robotic System Integration
FMF-SLAM is designed for practical application. We integrated the system with a global positioning module (GNSS-RTK) and an Extended Kalman Filter (EKF) to fuse visual-depth data with global coordinates. To minimize trajectory drift over long distances, we incorporate Global Bundle Adjustment (BA), ensuring the map remains consistent over time.
Experimental Evaluation
Datasets and Benchmarks
We evaluated FMF-SLAM using a combination of standard benchmarks and custom real-world data:
- TartanAir & TUM: We focused specifically on the "challenging" subsets of TartanAir, which include fast motion, dimmed lighting, and night scenes.
- Realtest-GT & Realtest-Visual: These custom datasets were recorded using a RealSense L515 camera. To provide a rigorous ground truth for indoor tests, we utilized a high-precision laser tracker capable of millimeter-level accuracy.
Performance Results
FMF-SLAM demonstrates superior accuracy and robustness compared to both classical and modern learning-based SLAM systems:
- Accuracy: On the TartanAir dataset, FMF-SLAM achieved an Average Absolute Trajectory Error (ATE) of 1.07 cm, significantly outperforming DROID-SLAM (6.02 cm) and RESLAM (14.34 cm).
- Robustness: In dark scenarios, FMF-SLAM reduced localization errors by over 50% compared to its closest competitors. Notably, while other methods suffered from tracking failures (outliers) in extreme conditions, FMF-SLAM maintained a 100% success rate.
- Efficiency: The system processes frames at approximately 77.76 ms per frame on an Nvidia 3090, meeting the requirements for real-time operation.
Real-World Deployment
The system was successfully deployed on a security robot for outdoor testing. Over a 147-meter path, the system demonstrated an accumulation error of only 0.06%. To handle large-scale outdoor environments where active depth sensors like the L515 may struggle, the researchers utilized ZoeDepth, a pre-trained depth estimation model, to provide stable depth information for the robot trials.
Conclusion
FMF-SLAM represents a significant step forward in robust multimodal SLAM. By leveraging the efficiency of the Fourier Transform and the stability of multi-scale distillation, the system provides reliable localization in environments that typically blind visual sensors. Future work will explore the integration of LiDAR data and the application of Fourier-based encoders to other modalities, such as thermal imaging, to further enhance robotic perception in extreme environments like smoke-filled search-and-rescue zones.
Summary updated