TL;DR
Practical Image Coding Agents (PICA) is a training-free, multi-agent framework that makes existing image codecs "smart" by adapting them to diverse user requirements and degraded inputs. By treating codecs as black boxes and using a "schedule-execute-reflect" loop, PICA consistently outperforms standard and specialized codecs across perceptual, machine-vision, and standard compression scenarios without requiring any architectural redesign or retraining.
What Problem Does This Paper Address?
Traditional image codecs (like JPEG or H.266/VVC) and even modern neural image codecs (NIC) suffer from two major rigidities. First, they are typically optimized for a single objective—usually pixel-level fidelity (PSNR)—making them suboptimal for specialized tasks like human perceptual realism or downstream machine-vision tasks (e.g., object detection). Second, they assume "clean" input images. In reality, sensor noise or motion blur can confuse a codec, causing it to waste precious bits encoding artifacts rather than the actual signal.
Adapting a codec to these variables usually requires massive datasets and expensive retraining. The authors ask: Can we adapt existing, widely-used codecs to diverse scenarios and degraded inputs autonomously, without touching their internal architecture?
Methods
The authors introduce PICA, a framework comprising four collaborative agents that manage the compression pipeline through a Scenario-Guided Quality–Rate (SGQR) evaluation strategy.
- Perception Agent: Uses Vision-Language Models (VLMs) to interpret user intent (e.g., "compress for object detection") and analyze the image for degradations and Regions of Interest (ROIs).
- Improvement Agent (Pre-coding): Performs Quality-Rate (QR)-aware restoration. It doesn't just "fix" the image; it evaluates whether denoising or deblurring will actually improve the final bitrate-to-quality trade-off.
- Coding Agent: This is the core of the framework. It performs a global search for the best quality settings and utilizes a novel Agentic ROI Coding (AROIC) mechanism. AROIC treats the codec as a black box, cropping ROIs, encoding them with higher quality, and reassembling the image. This brings ROI capabilities to codecs that don't natively support them.
- After-Coding Improvement Agent: A final pass to clean up residual compression artifacts or degradations using quality-only metrics.
The system operates on a reflection loop: it proposes a coding plan, executes it using the target codec, evaluates the result against scenario-specific metrics (like LPIPS for humans or mAP for machines), and iterates until the goal is met.
Datasets
The framework was evaluated on three major benchmarks, with synthetic Gaussian noise and defocus blur added to test robustness:
- Kodak: 24 natural images for standard and perceptual evaluation.
- CLIC2020: 102 high-resolution images for varied visual content.
- HEVC (Classes C and D): Used for machine-vision evaluation, specifically object detection performance using the SFU-HW-Objects-v3 annotations.
Key Insights and Ideas
- The "Black Box" Advantage: By treating the codec as a black box, PICA is future-proof. It can be wrapped around H.266, a new neural codec, or even a legacy standard like JPEG without modification.
- Restoration as a Bitrate Saver: A key "aha" moment is that pre-coding restoration (like denoising) isn't just for aesthetics. By removing noise before the codec sees the image, PICA prevents the codec from wasting bits on high-frequency "junk," significantly improving compression efficiency.
- Agentic ROI Coding (AROIC): Most ROI methods are baked into the codec's math. AROIC proves you can achieve similar results through agentic scheduling—deciding which parts of the image to crop and how to independently encode them for the best aggregate result.
- Scenario-Specific Metrics: PICA changes its "definition of success" based on the user. For machine vision, it optimizes for task accuracy; for humans, it optimizes for perceptual metrics like DISTS and LPIPS.
Key Results
PICA was tested using both the conventional H.266/VVC and the neural Cheng2020 codec as base engines.
- Perceptual Compression: On the Kodak dataset, PICA achieved average bitrate savings of 34.5% to 62.6% over the baseline Cheng2020 codec at matched perceptual quality (LPIPS).
- Machine Vision: For object detection on HEVC Class D, PICA showed relative gains in mAP (mean Average Precision) of up to 80.4% compared to standard VVC at the same bitrate.
- Robustness: PICA outperformed specialized codecs like HiFiC (perceptual) and DenoiseCompression (noisy inputs) when those codecs were forced to operate outside their specific training niches.
- VLM Agnostic: The framework proved robust across different VLM backbones, including Qwen2.5-VL and GPT-4o, yielding nearly identical performance gains.
Limitations and Future Work
The primary limitation is computational overhead. Because PICA uses an iterative "schedule-execute-reflect" loop involving multiple encodings and VLM inferences, it is currently slower than a single-pass codec.
Future work suggested by the authors includes:
- Optimizing the efficiency of the agentic loops for real-time applications.
- Extending the framework to video compression, where temporal consistency adds a new layer of complexity.
- Exploring more sophisticated VLM-based feedback to further refine regional bit allocation.
Summary updated