Abstract
Autonomous driving has made great strides thanks to artificial intelligence, with numerous advanced methods proposed for vehicle-end target detection. However, the complexity of real traffic situations necessitates examining how to use these methods in real road conditions. This paper proposes RMMDet, a road-side multitype and multigroup sensor detection system. Using a ROS-based virtual environment, we simulate real-world conditions and implement multi-type sensor detection and multi-group sensor fusion, including camera-radar and camera-lidar detection based on result-level fusion. We produce local datasets and a physical sand table field for experimental validation. Furthermore, we link a multi-agent collaborative scheduling system to the fusion detection system, forming a complete roadside detection framework. Experiments demonstrate that RMMDet plays an important role in vehicle-road collaboration and optimization.
Introduction
While autonomous vehicle technology has advanced rapidly, individual vehicles remain limited by their onboard sensors' range and line-of-sight obstructions. Roadside Units (RSUs) offer a solution by providing a persistent, wide-area perspective of traffic. However, building an effective roadside system is challenging: it must handle diverse sensor types (multitype), manage multiple sensor clusters across different road segments (multigroup), and remain robust even if one sensor fails.
Current research often focuses on "feature-level" fusion, which can be computationally heavy and prone to total system failure if one data stream is corrupted. RMMDet aims to bridge this gap with a modular, efficient, and infrastructure-centric detection and scheduling system.
System Architecture
The RMMDet framework is organized into a three-layer architecture designed for modularity and robustness.
- Sensor Layer: Incorporates Lidar, Camera, and mmWave radar.
- Fusion Perception Layer: Implements result-level fusion. Instead of merging raw data, the system runs independent detectors (e.g., YOLOv4 for 2D images and PointPillars for 3D Lidar) and fuses the outputs. This ensures that if one sensor fails, the others can still provide critical data.
- Decision Planning Layer: A multi-agent scheduling system that uses fused data to build real-time traffic density maps and optimize vehicle routing.
Simulation and Experimental Setup
To validate the system, the authors developed a high-fidelity virtual environment using ROS (Robot Operating System) and Gazebo. This environment simulates complex road geometries including intersections, ring roads, and viaducts.
The study utilized several custom-generated datasets to train and validate the models:
- Image Dataset: 13,985 images generated via Blender and Gazebo, covering various perspectives and lighting conditions.
- Lidar Dataset: 1,750 frames of 3D point cloud data annotated in KITTI format.
- Physical Validation: A real-world 8m x 10m "sand table" environment (1:18 scale) using AWS Deepracers was used to test the "sim-to-real" transferability of the system.
Key Insights
- Modularity as Robustness: By choosing result-level fusion over feature-level fusion, the authors prioritize system uptime. The ability to maintain partial perception during a sensor failure is a critical safety feature for roadside infrastructure.
- Bridging Sparse and Dense Data: The "Radar ROI" approach handles the sparsity of radar data by expanding points into Regions of Interest based on known target sizes, making it easier to correlate with dense camera pixels.
- Functional Decomposition: Dividing the traffic environment into agents based on road geometry (e.g., an "Intersection Agent") allows the scheduling system to scale effectively.
Results
The system achieved high precision in simulation, with a pose error of less than 0.045m. Bird's-Eye View (BEV) IOU scores were consistently above 0.7, and Orientation Similarity (AOS) exceeded 0.9.
In traffic optimization tests with 10 autonomous vehicles, the multi-agent scheduling system increased total traffic flow from 624 to 668 vehicles (a ~7% improvement) by actively reducing congestion at intersections. The entire pipeline reached a processing speed of approximately 10 FPS, which is sufficient for real-time roadside-to-vehicle collaboration at this scale.
Conclusion and Future Work
RMMDet demonstrates a comprehensive roadside sensing framework that improves traffic flow and robust object detection. The primary bottleneck for system speed is currently time synchronization between the three sensor types. Future work will focus on improving synchronization algorithms to boost FPS and integrating more adaptive fusion methods that can better handle diverse real-world weather and lighting conditions.
Summary updated