When humans look at a heavily compressed image, we might notice blocky artifacts or blurred edges, but we can still easily identify the main subject. Traditional compression algorithms exploit this by discarding high-frequency information that the human eye struggles to perceive. However, deep learning models do not "see" the way humans do, and different machine learning tasks rely on entirely different granularities of information. A uniform compression strategy that works perfectly for one task can completely degrade the performance of another.
The Granularity Gradient of Downstream Tasks
To understand why compression affects models unevenly, we must look at the information requirements of different downstream tasks. Machine learning objectives exist on a gradient from coarse-grained (global) to fine-grained (localized or pixel-level).
At the coarse end of the spectrum, models require very little information to succeed. For instance, in natural language processing, Hayato Tsukagoshi et al. demonstrate in Redundancy, Isotropy, and Intrinsic Dimensionality of Prompt-based Text Embeddings that for text classification, up to 99.5% of prompt-based embedding dimensions can be discarded with almost no loss in performance. A similar phenomenon occurs in vision-language models. Tianfan Peng et al. show in Can visual input Be Compressed? A visual input token compression benchmark for large multimodal models that global tasks like instruction following tolerate aggressive visual token pruning, whereas fine-grained tasks like Optical Character Recognition (OCR) degrade severely because they depend on precise textual details.
As we move toward dense spatial predictions in computer vision, this sensitivity escalates dramatically. Max Ehrlich et al. establish in Analyzing and Mitigating JPEG Compression Defects in Deep Learning that lossy compression artifacts degrade dense prediction tasks like semantic segmentation far more severely than standard image classification. The authors of A Multi-Task Supervised Compression Model for Split Computing confirm this vulnerability, noting that pixel-level prediction tasks are inherently more susceptible to accuracy degradation from lossy representations than image-level or bounding-box level tasks.
The Mechanism of Feature and Frequency Mismatch
Why do dense prediction tasks break down under standard compression? The answer lies in the mismatch between what traditional compression discards and what localized machine tasks need.
Image classification models primarily rely on low-frequency, global semantic structures to determine what object is present. In contrast, tasks like object detection and semantic segmentation must determine exactly where boundaries lie. Zihao Liu et al. explain in DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework that standard JPEG compression discards high-frequency components that are vital for spatial localization and boundary definition. This frequency misalignment is also evident in 3D medical imaging; Tao Liu et al. show in Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds that the frequency priorities required for boundary detection during segmentation do not align with those needed for classification or anomaly detection.
[Raw Data] ───> [Lossy Compression] ───> Discards High-Frequencies & Fine Details
│
┌────────────────────────┴────────────────────────┐
▼ ▼
[Global/Coarse Tasks] [Dense/Localized Tasks]
(e.g., Image/Text Classification) (e.g., Segmentation, OCR, Detection)
Succeeds on low-frequencies Fails due to loss of spatial boundaries
Robust to Loss Highly Sensitive to Loss
When models are optimized for efficiency via pruning or quantization, they encounter similar hurdles. Jian Cheng et al. observe in Recent Advances in Efficient Computation of Deep Convolutional Neural Networks that localized tasks exhibit a much higher sensitivity to precision loss and structural changes than image classification.
This problem becomes even more complex when compressing intermediate neural network features rather than raw pixels. Guosheng Lin demonstrates in Compressed Feature Quality Assessment: Dataset and Baselines that tasks requiring the simultaneous preservation of high-level categorical information and precise spatial coordinates generate highly complex, non-linear distortion patterns under compression. As a result, traditional signal-fidelity metrics like Mean Squared Error (MSE) fail to capture the actual semantic information loss experienced by the model.
Designing Task-Aware Compression Systems
Recognizing that compression sensitivity is task-dependent allows researchers to move away from uniform, human-centric codecs and design machine-centric alternatives.
One approach is to optimize compression algorithms directly for the downstream model's objective. Benoit Brummer et al. demonstrate in Adapting JPEG XS gains and priorities to tasks and contents that tuning the internal weights of the JPEG XS codec specifically for semantic segmentation can reduce data needs by 59% compared to standard configurations. This highlights a core insight formalized by João Maria Janeiro et al. in Are Visual Recognition Models Robust to Image Compression?: visual quality for human perception does not correlate with feature utility for machine learning models, meaning compression strategies must be tailored to the specific downstream task.
This task-specificity also redefines how we scale training datasets under storage constraints. Katherine L. Mentzer et al. prove in Scaling Training Data with Lossy Image Compression that when storage is limited, training on a larger number of lossily compressed images outpaces training on fewer high-quality images. However, the authors emphasize that the optimal compression threshold is strictly task-dependent: small object detection fails under aggressive bit-depth reductions that are perfectly acceptable for coarse-grained image classification.
Ultimately, managing these trade-offs requires precise evaluation frameworks. As Kristian Fischer 0001 et al. note in Evaluation of Video Coding for Machines without Ground Truth, tasks requiring pixel-perfect accuracy are highly vulnerable to compression artifacts compared to bounding-box detection. To address this in specialized domains like autonomous driving, Madhumitha Sakthi et al. introduce in Impact of Video Compression Artifacts on Fisheye Camera Visual Perception Tasks localized zonal metrics to accurately quantify how compression degrades spatially dense perception tasks like depth estimation and segmentation across distorted visual regions.
Key Foundational Work
Recent Advances in Efficient Computation of Deep Convolutional Neural Networks (2018) — contributed 2 concept elements to this direction.
DeepN-JPEG: A Deep Neural Network Favorable JPEG-based Image Compression Framework (2018) — contributed 1 concept elements to this direction.
Machine Vision Guided 3D Medical Image Compression for Efficient Transmission and Accurate Segmentation in the Clouds (2019) — contributed 1 concept elements to this direction.