For decades, image compression algorithms like JPEG, BPG, and JPEG2000 have been engineered around a single target: the Human Visual System (HVS). Because human eyes are largely insensitive to high-frequency noise and subtle texture variations, classical codecs aggressively discard this information to save bits. This strategy works beautifully for human consumption, but it introduces a fundamental misalignment when the primary consumer of an image is a machine learning model.
As Xin Jin et al. explore in Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification, human-centric compression optimizes for visual fidelity, whereas machine models require semantic information. When we force artificial intelligence to process images compressed for humans, we are effectively blinding them to the features they need most, while simultaneously wasting bandwidth on aesthetic textures that machines completely ignore.
Why Machines and Humans See Differently
To understand why this misalignment happens, we must look at what deep neural networks actually exploit to make decisions. Humans prioritize low-frequency structural layouts and smooth, aesthetically pleasing textures. In contrast, computer vision models often rely on high-frequency details, precise pixel boundaries, and specific feature distributions that are imperceptible or irrelevant to us.
This divergence becomes critical across several distinct domains:
- Medical Imaging and Connectomics: In 3D medical segmentation or electron microscopy, identifying the exact boundary of a tumor or a neuron is paramount. Tao Liu et al. demonstrate in Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds that deep learning models rely heavily on high-frequency details that standard human-centric compression discards. Similarly, Core Francisco Park et al. developed EM-Compressor: Electron Microscopy Image Compression in Connectomics with Variational Autoencoders specifically to preserve these critical high-frequency structures for automated neuron segmentation.
- Text and Document Analysis: For Optical Character Recognition (OCR), the sharp edges of text characters represent high-frequency signals. Standard compression smooths these out to make backgrounds look clean to a human. Alessandro Gnutti et al. show in TFIC: End-to-End Text-Focused Image Compression for Coding for Machines that optimizing compression for machine readability over human aesthetics can drastically improve OCR accuracy at low bitrates, sometimes even outperforming OCR on uncompressed originals.
- Multimodal Reasoning: Modern Multimodal Large Language Models (MLLMs) use subtle, non-perceptual details for complex semantic reasoning. The authors of When MLLMs Meet Compression Distortion: A Coding Paradigm Tailored to MLLMs show that human-centric codecs degrade MLLM performance because they strip out these features. Binzhe Li et al. echo this in High Efficiency Image Compression for Large Visual-Language Models, noting that traditional compression discards textures vital for feature extraction in large-scale models.
Adapting Classical Codecs for Machines
How can we bridge this gap without inventing entirely new image formats from scratch? One powerful approach is to modify the quantization step of standard codecs like JPEG.
In standard JPEG, a global quantization table penalizes high frequencies because humans do not notice their absence. Zihao Liu et al. introduce DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework, which redesigns this quantization matrix specifically to preserve the features neural networks use for classification. Similarly, Cheng Zhang et al. show in Countering Adversarial Examples: Combining Input Transformation and Noisy Training that redesigning JPEG compression to favor neural networks can preserve clean-image accuracy while simultaneously defending against adversarial attacks.
Taking this further, Jin Young Choi 0002 et al. propose a Task-Aware Quantization Network for JPEG Image Compression that dynamically predicts image-specific quantization tables optimized for downstream tasks like classification or captioning, while remaining fully compatible with standard JPEG decoders.
Standard JPEG: [Image] -> [Human-Centric Quantization] -> Discards High Frequencies -> Blinds AI
Machine JPEG: [Image] -> [Task-Aware Quantization] -> Preserves Semantic Edges -> Fosters AI Accuracy
Alternatively, we can use neural preprocessing to filter out human-centric redundancies before the image ever hits a standard codec. Guo Lu et al. implement this in Preprocessing Enhanced Image Compression for Machine Vision via a Neural Preprocessing Module (NPP). The NPP strips away aesthetic textures that consume bits but offer no semantic value to an AI, resulting in images that look highly distorted to humans but are highly efficient for machine vision.
Shifting from Pixels to Semantic Latents
If an image is only meant to be read by a machine, reconstructing pixels at all becomes an unnecessary bottleneck. Instead, we can compress the meaning of the image directly within a model's latent space.
Ruiqi Shen et al. demonstrate this paradigm shift in Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models by quantizing and compressing CLIP features directly rather than pixel data, achieving massive bitrate savings. For downstream language tasks, Chia-Hao Kao demonstrates in Bridging Compressed Image Latents and Multimodal Large Language Models that feeding compressed image latents directly into MLLMs bypasses full image reconstruction entirely, cutting decoding complexity by 95%.
This paradigm aligns closely with split computing, where a mobile device runs a lightweight encoder and transmits intermediate features to the cloud. Yoshitomo Matsubara et al. formalize this in the SC2 Benchmark: Supervised Compression for Split Computing, proving that general-purpose compression wastes bandwidth encoding visual details that do not contribute to downstream model performance. However, operating directly in the compressed domain is challenging; as Rémi Piau et al. note in Learning on Entropy Coded Images with CNN, the final stage of entropy coding destroys spatial structures, creating a fundamental tension between removing statistical redundancy and maintaining extractable features.
To guide what is worth compressing in these latent spaces, researchers use foundation models as semantic filters. Jinming Liu et al. introduce SDComp in Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs, using Large Multimodal Models to rank objects by semantic importance and allocate bits accordingly. This ensures that the discriminative power of features is preserved where it matters most—an objective formalized by Changsheng Gao et al. in DMOFC: Discrimination Metric-Optimized Feature Compression. Similarly, Zhibo Chen et al. apply this to facial images in Learning based Facial Image Compression with semantic fidelity metric, prioritizing face verification accuracy over pixel-perfect reconstruction.
Balancing the Human-Machine Trade-off
Can a single compression stream satisfy both worlds? In many practical scenarios, like cloud-based surveillance or autonomous driving, an image might be analyzed by an AI in real-time but occasionally inspected by a human operator.
/---> Base Layer --------> Machine Vision (Object Detection)
[Latent Space] ---
\---> Enhancement Layer -> Human Viewer (Visual Fidelity)
To address this, Hyomin Choi et al. propose a scalable approach in Scalable Image Coding for Humans and Machines. They partition the latent space into a base layer optimized for machine vision and an enhancement layer that restores full visual fidelity for human viewers.
Other approaches adapt existing networks dynamically. Yi-Hsin Chen et al. introduce TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception, which uses learnable prompts to shift a pre-trained codec's focus from human perception to machine perception without altering the underlying weights. Finally, Ruoyu Feng et al. attempt to bridge this gap entirely in Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior, using generative priors to satisfy the human demand for sharp textures while preserving the precise feature distributions required by machine vision algorithms.
Key Foundational Work
Learning based Facial Image Compression with semantic fidelity metric (2019) — contributed 1 concept elements to this direction.
DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework (2018) — contributed 1 concept elements to this direction.
Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds (2019) — contributed 1 concept elements to this direction.