Image Coding for Joint Human and Machine Vision

This direction is split into other directions: human-centric compression bias (perceptual-semantic misalignment). Its last version stays readable and citable here.

Version 1 of 1162 papers196 conceptsfrozen 2026-10-04

For decades, image and video compression standards like JPEG, HEVC, and VVC have been designed with a single consumer in mind: the human eye. These codecs are optimized to minimize "perceptual distortion," discarding data that humans cannot see while preserving smooth gradients and natural textures.

However, in the era of the "AI of Things" (AIoT) and autonomous systems, the primary consumers of visual data are increasingly machines—Deep Neural Networks (DNNs) and Multimodal Large Language Models (MLLMs). As researchers like Sien Chen and Zhengguang Li argue in A New Image Codec Paradigm for Human and Machine Uses, what is "perceptually important" to a human is often "semantically redundant" for a machine, and vice versa.

The Perceptual Misalignment

The fundamental problem is a misalignment of utility functions. Human-centric compression prioritizes pixel-level fidelity and aesthetic quality, often using metrics like Peak Signal-to-Noise Ratio (PSNR) or MS-SSIM. To save bits, these codecs might blur high-frequency details or smooth out textures.

To a machine vision model, these "invisible" details are often the most critical. Maurice Weber and colleagues demonstrated in Lossy Image Compression with Recurrent Neural Networks that machine-optimized compression often preserves sharp edges and high-frequency patterns that appear as "noise" to humans but are essential for classification accuracy. Conversely, Jonas Löhdefink showed in the context of autonomous driving that a higher PSNR does not necessarily lead to better semantic segmentation; in fact, standard human-centric artifacts can disproportionately degrade AI performance (GAN- vs. JPEG2000 Image Compression for Distributed Automotive Perception).

Feature-Centric Compression

Rather than compressing pixels, a new wave of research proposes compressing the features or latents that machines actually use.

In Image Coding for Machines with Omnipotent Feature Learning, Ruoyu Feng and Xin Jin introduce the concept of "omnipotent features." Instead of a codec producing an RGB image that is then fed into a vision model, the codec produces a compact, universal representation. This representation is designed to be "machine-native," allowing multiple downstream tasks—like detection, segmentation, and recognition—to run directly on the compressed bitstream without full decompression.

This shift allows for massive efficiency gains. For example, Manabu Nakanoya and Sandeep Chinchali developed TASKNET, which enables robots to compress sensory data by up to 11x more than standard methods by focusing only on task-relevant features (Task-relevant Representation Learning for Networked Robotic Perception).

Scalable Bitstreams for Dual Use

In many real-world scenarios, such as smart cities or medical imaging, data must serve both masters: an AI must flag an event, and a human must eventually review it. This has led to the development of Scalable Image Coding (SIC).

Hyomin Choi and Ivan V. Bajic pioneered a "latent-space scalability" approach in Scalable Image Coding for Humans and Machines. Their framework partitions the compressed data into:

  1. A Base Layer: Containing the essential semantic features for machine vision.
  2. An Enhancement Layer: Containing the residual texture information needed to reconstruct a high-quality image for human viewing.

This "one-bitstream-two-uses" philosophy is further refined by Shiqi Wang in Towards Analysis-friendly Face Representation, where deep features provide the foundation for face verification, while texture residuals allow for human-level inspection.

Bypassing the Decoder for MLLMs

The most recent evolution of this idea targets Multimodal Large Language Models (MLLMs). Standard pipelines require decompressing an image into a massive RGB grid, only for the MLLM's vision encoder (like CLIP) to immediately compress it back into tokens.

Researchers like Chia-Hao Kao propose bypassing this "pixel bottleneck" entirely. In ComNeck: Bridging Compressed Image Latents and Multimodal LLMs, they introduce a "transform-neck" that maps the compressed latents of a neural codec directly into the LLM's embedding space. By avoiding the reconstruction of raw pixels, this method achieves up to 80% bitrate savings and a 95% reduction in decoding complexity while maintaining high accuracy on complex reasoning tasks.

Similarly, Jingming Liu and Bin Li developed CoTAM, which uses shallow CLIP-based guidance to ensure that the compression process preserves the specific visual cues—like those needed for OCR or counting—that MLLMs rely on (When MLLMs Meet Compression Distortion).

Generative Priors as the Bridge

A final emerging mechanism for harmonizing these two visions is the use of generative priors. If a machine only needs a sparse semantic map to understand a scene, can we "hallucinate" the rest for the human viewer?

Frameworks like SLIM (Semantic-based Low-bitrate Image compression for Machines) and Diff-ICMH (Harmonizing Machine and Human Vision in Image Compression with Generative Prior) use diffusion models to reconstruct realistic textures from ultra-low-bitrate semantic features. This allows the system to transmit only the "meaning" of the image, relying on the decoder's generative power to make the result visually pleasing to humans, effectively decoupling semantic accuracy from perceptual realism.

Papers in this version

Showing 60 of 162 papers.

History & citationVersion 1 · 2026-10-04

Split into other directions · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_image_coding_for_joint_human_and_machine_c1d75c85,
  title = {Image Coding for Joint Human and Machine Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-coding-for-joint-human-and-machine-vision/rdr_c1d75c85d91b4ac8905d6e584980432e},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_image_coding_for_joint_human_and_machine_c1d75c85_v1,
  title = {Image Coding for Joint Human and Machine Vision},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/image-coding-for-joint-human-and-machine-vision/rdr_c1d75c85d91b4ac8905d6e584980432e/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}