Abstract
Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than 1000× on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.
Introduction
The explosion of data from sources like autonomous vehicles and climate simulations has made storage and processing increasingly difficult. Traditional lossy compression standards, such as JPEG, are designed to preserve perceptual fidelity—how an image looks to a human eye. However, much of modern data is processed exclusively by downstream algorithms. This "perceptual overhead"—bits used to maintain sharp textures or specific color shades—is often irrelevant for tasks like object detection or classification. This paper introduces a framework for lossy compression that preserves only the information necessary for lossless prediction on specific tasks, discarding the rest to achieve massive storage gains.
Theoretical Framework: Maximal Invariants
The core of this approach is the concept of the maximal invariant. If a set of tasks is invariant to certain transformations (e.g., rotation, scaling, or specific data augmentations), the maximal invariant is a representation that retains only the information that changes across classes while discarding everything else.
By formalizing "lossless prediction" through group invariance, the authors prove that optimal task-centric compression involves becoming perfectly invariant to the "nuisances" the task ignores. This creates a linear relationship between bits saved and "excess risk," allowing for a predictable trade-off between storage and accuracy.
Proposed Methods
The authors propose several practical implementations of this theory:
- Variational Invariant Compressor (VIC): An encoder-decoder model trained to reconstruct the original, unaugmented version of an image from an augmented input. This forces the encoder to ignore augmentation-induced "noise."
- Bottleneck InfoNCE (BINCE): A contrastive approach that avoids high-dimensional pixel reconstruction. It learns to identify if two representations originate from the same source image while using an "entropy bottleneck" to enforce conciseness.
- Zero-Shot Compression with CLIP: Pre-trained models like CLIP, which already discard many nuisances through self-supervised learning, can be converted into compressors by adding a small entropy bottleneck to their feature representations.
Experimental Results
Evaluation across 10 diverse datasets (including ImageNet, medical PCam, and scientific Galaxy Zoo) demonstrated significant advantages:
- 1000× Bit-rate Savings: On ImageNet, the invariant compressor outperformed JPEG by over three orders of magnitude while maintaining high classification accuracy.
- Lossless Prediction: On the STL-10 dataset, the BINCE method achieved 121× better compression than PNG with zero decrease in test accuracy.
- Generalization: The CLIP-based compressor showed zero-shot success on 8 datasets, outperforming JPEG by 150× on the Pets dataset and 126× on Caltech-101.
Discussion and Limitations
While powerful, this approach has trade-offs. The primary "catch" is irrecoverable loss: once data is compressed for a specific set of invariant tasks, information needed for other tasks (like segmentation or reconstruction) is permanently discarded. Furthermore, methods like BINCE produce bitstrings that are not human-interpretable, making manual data inspection difficult. Finally, performance is highly dependent on the quality of user-defined augmentations. Future work aims to develop "foundation compressors"—universal models that can serve as a starting point for various task-centric compression needs.
Summary updated