Lossy Compression for Lossless Prediction

Yann Dubois, Benjamin Bloem-Reddy, Karen Ullrich, Chris J. Maddison

Introduction: The Perceptual Overhead

We are drowning in data. From climate sensors to autonomous vehicle fleets, we generate petabytes of information that no human will ever see. However, our standard compression tools—like JPEG or WebP—are designed to preserve "perceptual fidelity." They keep the details that make an image look good to a human observer.

For an AI performing a specific task, such as identifying a car or a tumor, most of those "pretty" pixels are noise. For example, a high-resolution photo of a dog might take 1 MB, but a detailed text caption describing the dog—which is often enough for a classifier—takes only 1 KB. This paper introduces a "task-centric" compression framework that discards task-irrelevant perceptual details, achieving over 1000× better compression than JPEG on ImageNet without sacrificing downstream classification accuracy.

Bridging Compression and AI

The authors tackle the challenge of building compressors that retain only the information necessary for algorithmic prediction, effectively throwing away the perceptual overhead. They bridge the gap between Rate-Distortion Theory (the mathematics of compression) and Invariant Representation Learning (the mathematics of AI tasks).

The Rate-Invariance Theorem

The foundation of this work is the Rate-Invariance (RI) theorem. It characterizes the minimum bits needed to perform well on any task that is "invariant" to certain transformations (like rotating or cropping an image). The theorem shows that the optimal strategy is to compress the data into its "maximal invariant"—a representation that completely ignores these transformations.

Illustration of mapping data to an invariant representation, discarding task-irrelevant variations.

Practical Compressors

Based on this theory, the authors propose several practical methods:

  1. Variational Invariant Compressor (VIC): A neural network that takes an "augmented" (transformed) image and tries to reconstruct a "canonical" (clean, unaugmented) version of it through a narrow bit-rate bottleneck.
A canonical, unaugmented reconstruction of a digit, demonstrating what the compressor retains.
  1. Bottleneck InfoNCE (BINCE): A contrastive learning approach. Instead of reconstructing pixels, it learns to recognize if two different augmented views came from the same original image. This avoids the need for a high-dimensional decoder.
  2. Zero-Shot CLIP Compressor: The authors demonstrate that pre-trained Self-Supervised Learning (SSL) models like CLIP can be turned into powerful, generic compressors simply by adding an "entropy bottleneck" to their representations.

Key Insights

  • Lossy Images, Lossless Prediction: You can lose a massive amount of visual data—making the image unrecognizable to humans—while keeping the prediction performance perfectly "lossless" relative to the original raw data.
  • Invariance is the Key: By defining what an algorithm shouldn't care about (e.g., slight rotations, lighting changes), we define exactly what the compressor can safely delete.
  • SSL Models are Natural Compressors: Models like CLIP are already trained to discard "noise" to find semantic meaning. Adding a bit-rate constraint allows them to function as high-performance, "zero-shot" compressors for datasets they've never seen during training.

Empirical Results

The researchers evaluated their methods across a wide variety of domains, from standard vision datasets (ImageNet, CIFAR10) to specialized tasks (PCam for biological imaging, Stanford Cars).

  • Massive Compression Gains: On ImageNet, their CLIP-based compressor achieved 1104× compression gains over JPEG while maintaining the same classification accuracy. On STL10, the BINCE model achieved 121× better compression than PNG with a 0% decrease in test accuracy.
  • Generality: The "zero-shot" compressor worked across 8 different datasets without being specifically re-trained for them, proving that task-centric compression can be broadly applicable.
  • Efficiency: The authors provided a script that can train a generic compressor in less than five minutes on a single GPU.

Limitations and Future Directions

While powerful, task-centric compression is a "one-way street." If you compress an image for classification and later decide you need to do fine-grained image segmentation, the necessary information may have already been irrecoverably discarded. Furthermore, the system's efficiency depends entirely on correctly identifying the transformations the task is invariant to; incorrect assumptions will lead to vital information loss.

Finally, while VIC produces "canonical" reconstructions that humans can somewhat interpret, BINCE produces abstract mathematical vectors that are entirely unreadable. Moving forward, the authors suggest that extending these "text-like" semantic representations to video could lead to even more staggering compression gains, potentially up to 1,000,000×.

Summary updated

Abstract

Most data is automatically collected and only ever "seen" by algorithms. Yet, data compressors preserve perceptual fidelity rather than just the information needed by algorithms performing downstream tasks. In this paper, we characterize the bit-rate required to ensure high performance on all predictive tasks that are invariant under a set of transformations, such as data augmentations. Based on our theory, we design unsupervised objectives for training neural compressors. Using these objectives, we train a generic image compressor that achieves substantial rate savings (more than 1000x on ImageNet) compared to JPEG on 8 datasets, without decreasing downstream classification performance.
BibTeX
@inproceedings{dubois2021lossy,
  author = {Yann Dubois and Benjamin Bloem-Reddy and Karen Ullrich and Chris J. Maddison},
  title = {{Lossy Compression for Lossless Prediction}},
  booktitle = {Advances in Neural Information Processing Systems},
  year = {2021},
  url = {https://openreview.net/forum?id=hPFbLyf9AYU},
}

Loading related content...

Figure