TL;DR
This paper demonstrates that for autonomous driving, traditional image compression metrics like Peak Signal-to-Noise Ratio (PSNR) are misleading. While GAN-based compression produces lower PSNR than JPEG2000 at ultra-low bitrates, it preserves the "global structure" necessary for machine perception, leading to significantly better semantic segmentation performance—especially when the perception model is retrained on compressed images.
What Problem Does This Paper Address?
Autonomous vehicles are data-hungry. A single car generates massive amounts of sensor data that must be transmitted across internal automotive bus systems (like CAN or Ethernet). These systems have limited bandwidth, creating a significant bottleneck.
To fit this data through the pipe, we must compress it. However, standard codecs like JPEG, JPEG2000, and WebP were designed to minimize pixel-wise distortion (Mean Squared Error) for human viewers. In a "distributed perception" setup—where a sensor compresses an image and a central unit performs semantic segmentation—pixel-perfect reconstruction might be less important than preserving the semantic "meaning" of the scene. The authors investigate whether modern Generative Adversarial Network (GAN) based compression is a better fit for these machine-centric tasks than traditional standards.
Methods
The authors compare four compression schemes: JPEG, JPEG2000, WebP, and a GAN-based autoencoder.
- The GAN Architecture: Based on the work of Agustsson et al., the generator is an autoencoder. The encoder downsamples the image by a factor of 16 and produces a compact latent representation. This representation is quantized to achieve the target bitrate. To make the quantization differentiable for training, the authors use a softmax-based approximation. The decoder then reconstructs the image using residual units and transposed convolutions.
- Training: The system uses a multi-scale PatchGAN discriminator. The loss function is a weighted sum of a standard GAN loss, a feature matching loss (comparing internal discriminator activations), and a pixel-level similarity loss (MSE).
- Perception Task: For the downstream task, they use DeepLabv3 with a ResNet38 backbone for semantic segmentation.
- The "Retraining" Strategy: A key part of their method involves retraining the segmentation model specifically on images that have already been processed by the GAN decoder, allowing the model to learn and ignore compression-specific artifacts.
Datasets
The study exclusively uses the Cityscapes dataset, a standard benchmark for urban semantic segmentation.
- Training: 2,975 annotated images.
- Validation: 500 annotated images (used for hyperparameter tuning).
- Testing: 1,525 images (results submitted to the official Cityscapes portal for evaluation).
Key Insights and Ideas
- The PSNR Paradox: At ultra-low bitrates (around 0.0625 bits per pixel), JPEG2000 achieves a PSNR nearly 6 dB higher than the GAN approach. To a human, the JPEG2000 image looks "closer" to the original. However, the GAN-compressed image results in better segmentation. This proves that distortion metrics (pixel-level) and perception metrics (structure-level) are often at odds.
- Global vs. Local: Traditional codecs often create "block artifacts" or blurriness that destroys the edges of objects. GANs, by nature, try to synthesize realistic-looking textures. Even if the GAN "hallucinates" the exact texture of a road or a building, it tends to preserve the sharp boundaries and global shapes that a segmentation network relies on.
- Artifact Adaptation: The most significant gains come from training the perception module "in the loop." By exposing the segmentation network to GAN-reconstructed images during training, the network learns to see through the specific types of distortions the GAN introduces, leading to a massive 20% absolute jump in mIoU.
Key Results
The authors evaluated the models at a highly constrained bitrate of 0.0625 bits per pixel (bpp):
- Distortion Metrics: JPEG2000 dominated with a PSNR of 34.03 dB vs. the GAN’s 27.73 dB.
- Standard Perception (No Retraining): Despite the lower PSNR, the GAN achieved a 50.4% mIoU on the test set, outperforming JPEG2000’s 46.0%.
- Optimized Perception (Retrained): When the segmentation model was fine-tuned on GAN-compressed images, the performance climbed to 63.0% mIoU.
- Comparison: For context, the segmentation model achieves 75.3% mIoU on uncompressed images. The GAN approach recovers a significant portion of that performance even at extreme compression levels.
Limitations and Future Work
- Small Object Recognition: The authors noted that while GANs are great at global structure, they can struggle with small, high-frequency details. For instance, traffic signs were often unrecognizable in GAN reconstructions, which is a critical failure point for autonomous driving.
- Computational Complexity: GAN-based encoding and decoding are significantly more computationally expensive than JPEG, which may pose challenges for real-time deployment on automotive hardware.
- Hardware Constraints: Due to GPU memory limits, images had to be downsampled by a factor of 2 during training.
- Future Directions: The authors suggest that end-to-end optimization—where the compression loss and the segmentation loss are optimized simultaneously—could yield even better results.
Summary updated