TL;DR
This paper introduces ECSIC, a novel deep learning method for stereo image compression that leverages "epipolar cross-attention" to exploit the strong correlation between stereo image pairs. ECSIC achieves state-of-the-art compression performance while maintaining fast encoding and decoding speeds, making it suitable for real-time applications.
What Problem Does This Paper Address?
Stereo image compression (SIC) aims to reduce the storage and transmission bandwidth required for pairs of images captured by stereo cameras. This is crucial for applications like autonomous driving and virtual reality streaming, which demand high compression rates and low latency. While traditional methods exist, learned compression techniques offer an advantage due to their potential for faster encoding and decoding. The core challenge in SIC is effectively exploiting the mutual information between the two images to achieve significant bitrate savings, especially given potential occlusions and differing viewpoints. Existing learned methods often rely on complex disparity estimation or autoregressive models, which can be computationally expensive or slow.
Methods
ECSIC is a learned compression method built upon a standard autoencoder architecture with a hyperprior for entropy modeling. Its novelty lies in two key components:
- Stereo Cross Attention (SCA) Module: This module allows the network to attend to corresponding epipolar lines in the stereo pair. By restricting attention to these lines (assuming rectified images), it significantly reduces computational complexity compared to full cross-attention, enabling parallel processing of image rows. This module is integrated into both the encoder/decoder and hyperprior branches.
- Stereo Context Modules (Cy and Cz): These modules are introduced to improve the entropy estimation for the right image by using the left image as context. This helps in more accurately predicting the probability distributions of the latent representations, leading to better compression.
The entire network is trained end-to-end to optimize a rate-distortion trade-off. Notably, ECSIC avoids autoregressive components, contributing to its fast decoding speed.
Datasets
The paper evaluates ECSIC on two popular stereo image datasets:
- Cityscapes: Contains stereo pairs of urban street scenes, with a large number of training, validation, and test images.
- InStereo2k: Features indoor scenes with objects closer to the camera.
The authors also mention that the method is end-to-end trainable on any stereo image dataset.
Key Insights and Ideas
- Epipolar Constraint for Efficiency: Restricting cross-attention to epipolar lines is a clever way to leverage stereo geometry for efficient information flow without the quadratic complexity of standard attention. This is a key enabler for their fast and effective method.
- Contextual Entropy Modeling for Stereo: Using one image's latent representation to inform the entropy model of the other image's latent representation is a powerful technique for exploiting stereo redundancy, especially when direct geometric warping is avoided.
- Balancing Performance and Speed: ECSIC demonstrates that state-of-the-art compression performance in stereo can be achieved without resorting to slow autoregressive models, making it practical for real-time applications.
Key Results
- State-of-the-Art Performance: ECSIC achieves state-of-the-art performance on both the Cityscapes and InStereo2k datasets, outperforming other learned and traditional codecs in terms of PSNR and bitrate savings.
- Competitive with VVC: On Cityscapes, ECSIC surpasses VVC (a high-performance traditional codec) at bitrates above 0.15 bpp.
- Fast Encoding/Decoding: The method exhibits significantly lower encoding and decoding times compared to other learned and traditional methods, even outperforming them on the InStereo2k dataset.
- Component Effectiveness: Ablation studies confirm the significant contributions of the SCA modules and stereo context modules to the overall performance. Removing context modules leads to a notable drop in performance.
Limitations and Future Work
- Generalization to Different Camera Setups: The current evaluation is limited to specific datasets. Generalizing to diverse camera setups might require larger and more varied training data.
- Rectified Images Assumption: The SCA module relies on the assumption that input images are rectified.
- Hardware Acceleration: Achieving the reported fast encoding/decoding times requires a GPU or similar neural hardware accelerator.
- Future Work: The authors suggest that further improvements could be made by exploring more diverse training datasets and potentially incorporating more advanced attention mechanisms or context modeling techniques.
Summary updated