Abstract
Autonomous navigation in unstructured environments—such as disaster zones or natural caves—presents unique challenges including irregular geometries, poor lighting, and sensor noise. This paper presents the Navigation Multimodal Fusion Network (NMFNet), a framework designed to handle these complexities by fusing RGB, Lidar, and 3D point cloud data. Integrated within the Robot Operating System (ROS), NMFNet utilizes a three-branch architecture to extract and combine features for robust steering prediction. To overcome the sim-to-real gap, the authors employ domain randomization during training in high-fidelity simulations. Experimental results demonstrate that NMFNet significantly outperforms single-sensor baselines, achieving a steering prediction Root Mean Square Error (RMSE) of 0.389 and successful real-world deployment on the BeetleBot platform.
Introduction
Traditional autonomous navigation systems are typically optimized for structured environments like well-lit hallways or paved roads. However, real-world applications such as search and rescue in collapsed buildings or cave exploration require robots to navigate "unstructured" settings characterized by irregular obstacles and unpredictable lighting. While end-to-end deep learning models offer a potential solution by mapping sensor data directly to motor commands, single-sensor models (e.g., RGB-only) often fail under varying environmental conditions. This research addresses these limitations by proposing a multimodal approach that ensures reliability through sensory redundancy.
Methodology: The NMFNet Framework
The core of the proposed system is the Navigation Multimodal Fusion Network (NMFNet). The architecture is designed to process diverse sensory inputs through a three-branch approach:
- RGB Branch: Utilizes a ResNet8 backbone to extract visual features from standard camera images.
- Distance Map Branch: Processes 2D Lidar-derived distance maps using a ResNet8 backbone to provide a 180-degree safety buffer.
- Point Cloud Branch: Employs a symmetric function (similar to PointNet) to interpret 3D spatial structures.
The framework employs a hybrid fusion strategy. "Early fusion" concatenates RGB and point cloud features, which share a common viewpoint, followed by $1 \times 1$ convolutional layers. "Late fusion" then integrates these with Lidar features to predict the final steering angle.
Data Acquisition and Simulation
To train the model, the authors generated a custom dataset of 120,000 synchronized records using the Gazebo simulator and ROS. The dataset covers four distinct environments: Normal City, Collapsed House, Collapsed City, and Natural Cave. To ensure the model generalizes to the real world, Domain Randomization was applied, varying textures and lighting conditions in simulation to prevent overfitting to specific visual details.
Results and Real-World Deployment
NMFNet demonstrated superior performance compared to established baselines. Quantitatively, it achieved an RMSE of 0.389 on steering prediction, whereas DroNet and Inception-V3 recorded 0.756 and 1.16, respectively. Qualitative analysis using Grad-CAM revealed that the fusion model focuses its attention on traversable paths, whereas single-sensor models often focus on irrelevant environmental noise.
The system was deployed on BeetleBot, a 6-wheeled robot equipped with a rocker-bogie suspension and an Nvidia Jetson embedded board. The robot successfully navigated real-world obstacles and rough terrain without additional fine-tuning, validating the effectiveness of the simulation-based training and domain randomization.
Conclusion and Future Work
While NMFNet provides a robust foundation for unstructured navigation, challenges remain regarding micro-obstacle detection and uncertainty estimation. Future research will focus on distinguishing traversable debris from impassable obstacles and integrating local avoidance with high-level goal-directed path planning.
Summary updated