TL;DR
Autonomous vehicles will need remote assistance for tricky situations, but sending massive sensor data (camera, lidar) in real-time is a bottleneck. This paper explores using deep generative models (like VAEs and GANs) to compress this data efficiently, showing that VAEs offer a good balance of compression and speed for real-time remote assistance, while GANs provide superior reconstruction quality.
What Problem Does This Paper Address?
As autonomous vehicles (AVs) become more sophisticated, they'll still encounter situations beyond their current capabilities. Remote assistance, where a human operator intervenes, is a crucial fallback. However, AVs generate vast amounts of sensor data (cameras, lidar) that need to be transmitted to the remote operator in real-time. This data volume can easily overwhelm network infrastructure, making efficient compression essential. While deep generative models have shown promise in compressing image and lidar data individually, their effectiveness and feasibility for the specific demands of AV remote assistance (balancing compression, reconstruction quality, and latency) haven't been thoroughly investigated.
Methods
The researchers evaluated state-of-the-art deep generative models for image and lidar compression. For image compression, they focused on Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs), specifically the models proposed by Ballé et al. [2] and Mentzer et al. [32], respectively. These models were trained to achieve a balance between rate (compression) and distortion (reconstruction quality), with an added perception metric for image quality.
For lidar compression, they adapted a VAE approach. The lidar data was pre-processed into a tensor representation that resembles image structure, allowing it to be compressed by the VAE.
To assess real-world applicability, they implemented an online, end-to-end compression pipeline using the Robot Operating System (ROS) and the CARLA simulator. This pipeline allowed them to measure processing times and latency for both image and lidar compression.
Datasets
- KITTI dataset: Used for training and evaluating image and lidar compression models. It contains RGB images and lidar scans.
- Waymo Open Perception Dataset: Used for training image compression models.
- CARLA simulator: Used as the sensor data source for the online pipeline and for simulating the AV environment.
Key Insights and Ideas
- GANs excel at image reconstruction quality, but VAEs are faster: While GANs produce visually superior reconstructions, especially at low bitrates, VAEs offer significantly lower processing times, making them more suitable for real-time remote assistance where latency is critical.
- Perception matters for remote assistance: Beyond just visual fidelity, the ability for a remote operator to perform tasks like object recognition on the reconstructed data is crucial. GANs showed better performance in object detection tasks on compressed images compared to VAEs, even at low bitrates.
- Lidar compression is challenging but promising: Adapting image compression techniques to lidar data is possible, but the pre-processing step to create a suitable tensor representation becomes a significant bottleneck in the online pipeline.
- End-to-end pipeline is key: Evaluating compression models within a realistic ROS-based pipeline is essential to understand their true performance and identify bottlenecks beyond just the model's inference time.
Key Results
- Image Compression:
- GANs achieved the best image reconstruction quality, even at very low bitrates (0.2-0.3 bpp), and maintained good object detection performance.
- VAEs offered a much better trade-off between processing time and reconstruction quality, achieving around 28 FPS for image compression with acceptable results.
- JPEG compression performed poorly at low bitrates, becoming unusable.
- Lidar Compression:
- The VAE approach for lidar compression showed promising results, with a mean Euclidean distance error of 0.3-0.5m at a bitrate of 1.8 bpp. However, typical lidar ring structures were lost, and noise increased.
- The lidar pre-processing step was identified as the main bottleneck in the online pipeline, limiting throughput to about 6 scans per second.
- Latency:
- The VAE-based image compression pipeline achieved a system latency of approximately 57.1 ms.
- The GAN-based image compression pipeline suffered from much higher latency (around 340.2 ms), making it unsuitable for real-time remote assistance.
Limitations and Future Work
- Limitations:
- The lidar pre-processing step in the online pipeline is a significant bottleneck.
- The current lidar compression approach results in a loss of fine details and structure.
- Perception analysis for lidar data was not performed due to limitations in reconstruction quality.
- The GAN approach, while providing better reconstruction, is too slow for real-time remote assistance.
- Future Work:
- Consider incorporating radar data for a more complete representation of the driving environment.
- Extend perception evaluation to other object classes (trucks, pedestrians, etc.).
- Explore more advanced lidar compression techniques, potentially adapting methods from other research.
- Optimize model structures and use acceleration methods to reduce inference times.
- Investigate combining camera and lidar data processing within a single pass.
- Optimize point cloud conversion methods and ROS messages to find a better balance between quality and pipeline speed.
- Deploy the implemented ROS framework on a test vehicle for real-world demonstration.
Summary updated