TL;DR
This paper proposes a novel approach to image compression for the web by leveraging generative AI to reconstruct images client-side using text prompts and additional conditioning inputs. This method achieves significant bandwidth savings (up to 99.8%) while preserving image meaning and structure better than traditional methods, potentially making the internet more affordable and accessible.
What Problem Does This Paper Address?
The internet is increasingly dominated by visual content, leading to massive data transfers and growing webpage sizes. Traditional image compression techniques, while reducing file sizes, often degrade image quality, especially at high compression ratios. This degradation negatively impacts user experience and contributes to the digital divide by making internet access more expensive, particularly in developing countries where affordable broadband is a challenge. The paper aims to address this by exploring how generative AI can reconstruct images at the "edge" (client-side), bypassing the need to transmit large image files altogether.
Methods
The core of the proposed solution is a framework that uses generative AI, specifically Stable Diffusion, to reconstruct images. Instead of sending the full image, the system sends "conditioning inputs" to the client. The paper explores a design space of these inputs, ranging from:
- Text-Only Prompts: Generating images from scratch based on textual descriptions.
- Edge Detection + Text Prompt: Using Canny edges to guide the AI on image structure, combined with text prompts for detail.
- Color Palettes + Edge Detection + Text Prompt: Adding a low-resolution color palette to ensure accurate color reproduction.
- Additional Conditioning Inputs + Text Prompt: Incorporating other inputs like semantic segmentation masks or in-painting masks for finer control and preservation of critical details (e.g., faces, text, logos).
The framework also utilizes JBIG2 compression for the generated Canny edge maps, further reducing their size. For evaluating image similarity, the paper moves beyond traditional pixel-level metrics like SSIM and PSNR, which can be misleading when AI reconstructs images, and instead uses the VGG16 convolutional neural network to compare high-level, holistic features.
Datasets
The paper uses two datasets for evaluation:
- Representative Dataset: Images scraped from the top 1000 globally visited websites (490 images total), categorized into seven types (e.g., E-commerce, News, Social Media).
- Synthetic Dataset: 150 high-resolution images collected from news websites and Wikipedia, chosen for their high quality and diversity.
Key Insights and Ideas
- Generative AI as a Compression Tool: The paper reframes generative AI not just for creation but as a powerful tool for compression by transmitting minimal inputs for client-side reconstruction.
- "Pseudo-Lossy" Compression: The proposed method is termed "Pseudo-Lossy" because it extracts semantic and structural information from the original image as conditioning inputs, guaranteeing the preservation of critical elements, unlike traditional lossy compression which irreversibly discards information.
- Conditioning Inputs Matter: The choice and combination of conditioning inputs (text, edges, color, masks) significantly impact the trade-off between bandwidth savings and perceptual similarity. More detailed inputs lead to better reconstruction but slightly lower bandwidth savings.
- VGG16 for Perceptual Similarity: Using VGG16 for feature comparison is crucial because pixel-level metrics can be fooled by AI's creative reconstruction, even if the semantic meaning is preserved.
- Human Preference: User studies indicate that reconstructed images, despite potential structural differences at the pixel level, are often preferred over traditionally compressed images due to the absence of compression artifacts.
Key Results
- Bandwidth Savings: The method achieves substantial bandwidth savings, with up to 99.8% in best-case scenarios (text-only prompts) and 92.6% on average with additional conditioning inputs.
- Perceptual Similarity: VGG16 scores indicate high preservation of image meaning and structure, with average scores of 0.82 for the most accurate strategy and 0.76 on average.
- User Study Findings:
- Respondents rated reconstructed images as more structurally similar to originals roughly half the time compared to traditional compression.
- Salient feature preservation was considered equally effective in maintaining webpage meaning.
- In 5 out of 8 tested images, users preferred the reconstructed images over compressed ones, even with equivalent bandwidth.
Limitations and Future Work
- Salient Feature Transmission Cost: While preserving salient features (faces, text, logos) is crucial, transmitting these unmodified can reduce overall bandwidth savings, especially if they constitute a large portion of the image.
- Generation Speed: Optimizing generation speed for real-time applications remains a challenge.
- Future Directions:
- Adaptive Compression: Dynamically choosing between traditional and AI-based methods based on image content and network conditions.
- Multi-modal Conditioning: Incorporating broader web page context (text, audio, video) to inform image reconstruction.
- Client-Side Reconstruction: Exploring the feasibility of running these models directly on user devices with the advent of lightweight language models.
- Regulatory and Ethical Considerations: Addressing privacy, image rights, bias in generative models, and developing standardization efforts.
Summary updated