Image Compression for Machine Perception

33 concepts31 papers1 proposal

For decades, image compression algorithms like JPEG, BPG, and JPEG2000 have been engineered around a single target: the Human Visual System (HVS). Because human eyes are largely insensitive to high-frequency noise and subtle texture variations, classical codecs aggressively discard this information to save bits. This strategy works beautifully for human consumption, but it introduces a fundamental misalignment when the primary consumer of an image is a machine learning model.

As Xin Jin et al. explore in Compression Tells Intelligence: Visual Coding, Visual Token Technology, and the Unification, human-centric compression optimizes for visual fidelity, whereas machine models require semantic information. When we force artificial intelligence to process images compressed for humans, we are effectively blinding them to the features they need most, while simultaneously wasting bandwidth on aesthetic textures that machines completely ignore.

Why Machines and Humans See Differently

To understand why this misalignment happens, we must look at what deep neural networks actually exploit to make decisions. Humans prioritize low-frequency structural layouts and smooth, aesthetically pleasing textures. In contrast, computer vision models often rely on high-frequency details, precise pixel boundaries, and specific feature distributions that are imperceptible or irrelevant to us.

This divergence becomes critical across several distinct domains:

Adapting Classical Codecs for Machines

How can we bridge this gap without inventing entirely new image formats from scratch? One powerful approach is to modify the quantization step of standard codecs like JPEG.

In standard JPEG, a global quantization table penalizes high frequencies because humans do not notice their absence. Zihao Liu et al. introduce DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework, which redesigns this quantization matrix specifically to preserve the features neural networks use for classification. Similarly, Cheng Zhang et al. show in Countering Adversarial Examples: Combining Input Transformation and Noisy Training that redesigning JPEG compression to favor neural networks can preserve clean-image accuracy while simultaneously defending against adversarial attacks.

Taking this further, Jin Young Choi 0002 et al. propose a Task-Aware Quantization Network for JPEG Image Compression that dynamically predicts image-specific quantization tables optimized for downstream tasks like classification or captioning, while remaining fully compatible with standard JPEG decoders.

Standard JPEG:  [Image] -> [Human-Centric Quantization] -> Discards High Frequencies -> Blinds AI
Machine JPEG:   [Image] -> [Task-Aware Quantization]    -> Preserves Semantic Edges  -> Fosters AI Accuracy

Alternatively, we can use neural preprocessing to filter out human-centric redundancies before the image ever hits a standard codec. Guo Lu et al. implement this in Preprocessing Enhanced Image Compression for Machine Vision via a Neural Preprocessing Module (NPP). The NPP strips away aesthetic textures that consume bits but offer no semantic value to an AI, resulting in images that look highly distorted to humans but are highly efficient for machine vision.

Shifting from Pixels to Semantic Latents

If an image is only meant to be read by a machine, reconstructing pixels at all becomes an unnecessary bottleneck. Instead, we can compress the meaning of the image directly within a model's latent space.

Ruiqi Shen et al. demonstrate this paradigm shift in Compression Beyond Pixels: Semantic Compression with Multimodal Foundation Models by quantizing and compressing CLIP features directly rather than pixel data, achieving massive bitrate savings. For downstream language tasks, Chia-Hao Kao demonstrates in Bridging Compressed Image Latents and Multimodal Large Language Models that feeding compressed image latents directly into MLLMs bypasses full image reconstruction entirely, cutting decoding complexity by 95%.

This paradigm aligns closely with split computing, where a mobile device runs a lightweight encoder and transmits intermediate features to the cloud. Yoshitomo Matsubara et al. formalize this in the SC2 Benchmark: Supervised Compression for Split Computing, proving that general-purpose compression wastes bandwidth encoding visual details that do not contribute to downstream model performance. However, operating directly in the compressed domain is challenging; as Rémi Piau et al. note in Learning on Entropy Coded Images with CNN, the final stage of entropy coding destroys spatial structures, creating a fundamental tension between removing statistical redundancy and maintaining extractable features.

To guide what is worth compressing in these latent spaces, researchers use foundation models as semantic filters. Jinming Liu et al. introduce SDComp in Tell Codec What Worth Compressing: Semantically Disentangled Image Coding for Machine with LMMs, using Large Multimodal Models to rank objects by semantic importance and allocate bits accordingly. This ensures that the discriminative power of features is preserved where it matters most—an objective formalized by Changsheng Gao et al. in DMOFC: Discrimination Metric-Optimized Feature Compression. Similarly, Zhibo Chen et al. apply this to facial images in Learning based Facial Image Compression with semantic fidelity metric, prioritizing face verification accuracy over pixel-perfect reconstruction.

Balancing the Human-Machine Trade-off

Can a single compression stream satisfy both worlds? In many practical scenarios, like cloud-based surveillance or autonomous driving, an image might be analyzed by an AI in real-time but occasionally inspected by a human operator.

                  /---> Base Layer --------> Machine Vision (Object Detection)
[Latent Space] ---
                  \---> Enhancement Layer -> Human Viewer (Visual Fidelity)

To address this, Hyomin Choi et al. propose a scalable approach in Scalable Image Coding for Humans and Machines. They partition the latent space into a base layer optimized for machine vision and an enhancement layer that restores full visual fidelity for human viewers.

Other approaches adapt existing networks dynamically. Yi-Hsin Chen et al. introduce TransTIC: Transferring Transformer-based Image Compression from Human Perception to Machine Perception, which uses learnable prompts to shift a pre-trained codec's focus from human perception to machine perception without altering the underlying weights. Finally, Ruoyu Feng et al. attempt to bridge this gap entirely in Diff-ICMH: Harmonizing Machine and Human Vision in Image Compression with Generative Prior, using generative priors to satisfy the human demand for sharp textures while preserving the precise feature distributions required by machine vision algorithms.

Key Foundational Work

Learning based Facial Image Compression with semantic fidelity metric (2019) — contributed 1 concept elements to this direction.
DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework (2018) — contributed 1 concept elements to this direction.
Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds (2019) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_human_centric_compression_bias_perceptua_8887b969,
  title = {human-centric compression bias (perceptual-semantic misalignment)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/human-centric-compression-bias-perceptual-semantic-misalignment/rdr_8887b969ceaf4cb8b1a9b5547400a32f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_human_centric_compression_bias_perceptua_8887b969_v1,
  title = {human-centric compression bias (perceptual-semantic misalignment)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/human-centric-compression-bias-perceptual-semantic-misalignment/rdr_8887b969ceaf4cb8b1a9b5547400a32f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake