Compressing Images for an Ensemble of Vision Encoders

Many vision-language systems now look at an image through several pretrained vision encoders at once. A typical mix is a CLIP-style encoder that understands what objects are, a convolutional network that notices small things, a segmentation model that tracks boundaries, and a document model that reads text. The extra encoders exist almost entirely to recover fine detail that the semantic encoder throws away. Yet the images these systems receive have nearly always passed through a lossy codec, and lossy compression removes high-frequency detail first: edges, small text, small objects. The likely result is that, at the bitrates used in practice, the detail encoders lose their signal before the semantic encoder does. The ensemble then collapses toward a single-encoder model while still paying for five forward passes.

This note proposes two steps. The first measures this collapse directly by tracing, for each encoder in a published ensemble and for the full multimodal model, how performance falls as bitrate drops. The second builds a learned image codec that spends bits so that no single encoder in the ensemble degrades much faster than the others. Its training loss measures distortion separately in each encoder's own feature space and always pushes hardest on whichever encoder is currently worst off. In two sentences: compression silently erases exactly the information that makes encoder ensembles worth running. We can measure how much, and we can train a codec that protects the "detail experts" by optimizing for the worst-served encoder rather than for pixels or for a single blended perceptual score.

Core Claim

Our claim has two parts. The first is a prediction that can fail. On detail-heavy benchmarks there is a bitrate below which a multi-encoder model's advantage over its own CLIP-only variant shrinks by more than half. The encoders that carry the advantage degrade first along the bitrate axis: the ConvNeXt branch, which Understanding Visual Detail Hallucinations identifies as most critical for small-object existence, and Pix2Struct for text.

The second part is a method claim. A codec trained with a worst-case, per-encoder feature distortion loss restores that advantage at equal bitrate. It does so better than a codec trained with a single fused perceptual metric, and better than simply teaching the multimodal model to tolerate compressed inputs.

We want feature distortion measured on paired original and reconstructed images. This choice also addresses the faithfulness problem that HiFiC documents. HiFiC trains a codec with an adversarial loss, so it is rewarded for producing textures that look natural, and it turns small text into "alien script." A paired feature-matching loss rewards reconstructing this image's text features, not plausible-looking glyphs.

Relation to Prior Work

Three groups of papers frame the proposal.

The ensemble papers motivate it. BRAVE fuses five frozen encoders through a small query transformer. Eagle concatenates encoder token grids channel-wise and reports Eagle-X5 at 598 on OCRBench. VisionWeaver routes between experts per image. Understanding Visual Detail Hallucinations ablates Eagle-X5 and shows its small-object gains come from specific detail encoders. All four evaluate on clean or naively resized images. None asks what happens after a codec.

The compression papers supply the method and constrain the design. Adapt-ICMH inserts small spatial and Fourier-domain adapters into frozen learned codecs and optimizes them for a downstream task, reaching an 88.57% BD-rate saving for ImageNet classification. We reuse its adapter recipe but replace the single task head with an ensemble of foundation encoders. Its result also warns us that a codec tuned for one task can discard what other tasks need. A-INN shows quantization is what destroys high-frequency subbands, which is why we expect the asymmetry we propose to measure. HiFiC supplies the generative-codec comparison and its documented hallucination failure.

The closest bridges are three papers, each of which we differ from in a stated way. DreamSim already blends DINO, CLIP and OpenCLIP into one perceptual distance (96.16% on NIGHTS versus 68.7% for LPIPS). It contains no detail or text experts, it was never used as a codec loss, and as a single fused score it lets a well-served encoder mask a badly served one. That masking is exactly what our worst-case weighting prevents. VLIC uses an external vision-language model as an offline preference judge for a diffusion codec. Its signal is a scalar human-alignment preference, not per-encoder fidelity for a downstream machine consumer. VTC compresses the tokens of a single encoder (DINOv2 or SAM3) as a 2D grid, achieving a 15.7x to 37.4x bitrate reduction. It is the natural alternative of transmitting features instead of pixels, and we treat it as a baseline rather than ignore it.

Research Program

Measurement study. We take the open Eagle-X5 ensemble (CLIP, ConvNeXt, SAM, EVA-02, Pix2Struct) and freeze it. We compress evaluation images with four codecs:

  • JPEG;
  • VVC through the VTM reference software;
  • an MSE-trained learned codec (cheng2020-anchor, one of the base codecs used by Adapt-ICMH);
  • HiFiC as the generative case.

Each codec is run at five to six bitrates spanning roughly 0.05 to 1 bits per pixel.

At each point we record two things. One is each encoder's normalized patch-grid feature distortion relative to the uncompressed image. The other is the full multimodal model's benchmark scores. We repeat the benchmark runs with each encoder ablated, following the ablation protocol of Understanding Visual Detail Hallucinations, so that the "ensemble advantage" can be tracked per encoder as a function of bitrate. For HiFiC we also measure whether detail encoders produce confident but wrong features on hallucinated texture, a failure mode that is worse than plain blur.

Codec. Starting from frozen cheng2020-anchor, we train Adapt-ICMH-style spatial and frequency adapters. Let $x$ be the image and $\hat{x}$ its reconstruction. Let $R$ be the estimated bitrate, $f_k$ the frozen patch-grid features of encoder $k$ among $K$ encoders, and $\lambda$ the rate trade-off. We define the relative distortion of encoder $k$ as

$$D_k = \frac{\lVert f_k(x) - f_k(\hat{x}) \rVert^2}{\lVert f_k(x) \rVert^2},$$

and train with

$$\mathcal{L} = R + \lambda \sum_{k=1}^{K} w_k D_k, \qquad w_k \propto \exp(\eta \bar{D}_k).$$

Here $\bar{D}_k$ is a running average of $D_k$ and $\eta$ sets how sharply the loss focuses on the worst-served encoder. As $\eta$ grows, this approaches minimizing the largest relative distortion, in the style of group-robust training; at $\eta = 0$ it is a uniform sum. A small pixel MSE term keeps reconstructions viewable by humans.

Training images are Flickr2W plus COCO. Text-rich images are scarce in both, so we treat text fidelity as a held-out stress test rather than a training target. If OCRBench lags, we will add text-heavy training images and report the result as a separate run.

Baselines

The do-nothing reference is the frozen Eagle-X5 fed images from standard VVC (VTM) at matched bitrate. This is what a deployed system sees today. Beating it means a positive BD-rate saving, measured on benchmark score versus bits per pixel, on OCRBench, POPE, MMVP and the small-object subset.

The ceiling is Eagle-X5 on uncompressed images, with its reported 598 on OCRBench and 1605 on MME Perception (Eagle). The floor is the CLIP-only variant under the same codec, which defines the ensemble advantage we try to preserve. We found no reported result on Lacuna for any multi-encoder model evaluated on compressed images at any bitrate, so every compressed-input number in this study is new and must be generated here.

The strongest existing method is Adapt-ICMH (paper) with the same adapter architecture, base codec, training data, number of steps and λ sweep, but trained against a single feature loss. The relevant reported figure is its 88.57% BD-rate saving on ImageNet classification. No figure exists for multimodal-model benchmarks. We run it with the CLIP feature loss, since CLIP is the encoder most systems rely on.

A second strong competitor is the same codec trained with DreamSim as the perceptual loss (DreamSim). It tests whether an existing ensemble metric already does the job.

HiFiC (paper) is the generative baseline. Its human-study figures (preferred at 0.237 bpp over BPG at 0.504 bpp) concern human preference, not machine consumers, so they set no target for us.

Two ablations isolate the key ingredient. The first trains with uniform weights ($\eta = 0$), so that worst-case weighting is the only difference from our method. The second drops each detail expert from the loss in turn, to test whether protecting ConvNeXt and Pix2Struct is what recovers small-object and OCR performance. We count the method as beating Adapt-ICMH and DreamSim-loss if it is better on at least the small-object and OCR benchmarks at matched bitrate and not worse on POPE.

Why not just…?

Why not just spend more bits? The comparison is always at equal bitrate, so "more bits" is not a competitor but the x-axis. The measurement study will show where on that axis the ensemble advantage disappears. If it only vanishes far below realistic bitrates, the method part of this proposal matters little, and we would report that.

Why not just fine-tune the multimodal model on compressed images? This is a real threat. If the information is still present but distorted, augmentation could recover it without touching the codec. We will fine-tune Eagle-X5's connector and encoders on VVC-compressed instruction data at mixed bitrates and compare. Our prediction is that augmentation helps at moderate bitrates but cannot restore detail that quantization has erased; A-INN's account of subband destruction suggests that information is genuinely gone. The two fixes are also complementary, so we will test them together.

Why not just use DreamSim, or LPIPS, as the codec's loss? DreamSim fuses CLIP- and DINO-family distances into one number and has no text or convolutional detail expert. A codec can therefore lower its loss mostly through the semantic encoders. The DreamSim-loss baseline and the $\eta = 0$ ablation directly test whether fusion and plain averaging are enough.

Why not just transmit encoder features with VTC instead of pixels? In a split-computing setting where the cloud only needs features, this could win. VTC reports large savings for a single encoder. For a five-encoder ensemble, however, it means running all five encoders on the edge device and sending five bitstreams. We will compare total bits and edge compute for per-encoder VTC streams against our single pixel stream. If VTC wins on bits, the honest conclusion is that pixel codecs suit only the case where the cloud runs the ensemble.

Proposed Experiments

The first decisive check is the crossover curve. For each codec, we plot ensemble-minus-CLIP-only score against bitrate on OCRBench, POPE, MMVP and the small-object split. Alongside it we plot per-encoder relative distortion $D_k$. The hypothesis fails if the advantage persists down to 0.1 bpp under VVC, or if the detail encoders do not degrade faster than CLIP.

The second check is whether worst-case weighting beats uniform weighting, single-encoder Adapt-ICMH and DreamSim-loss at matched rate, reported as BD-rate on each benchmark. It is backed by the per-encoder ablation showing which expert's protection produced the gain.

The third check probes faithfulness. On COCO, we compare object-existence F1 on small objects with the 73.7% LLaVA-1.5-7B small-object figure from Understanding Visual Detail Hallucinations, used only as a reference scale. We also count text and small-object hallucinations under HiFiC versus our codec at equal bitrate, to test whether paired feature matching avoids the invented-detail failure that adversarial codecs show.

Finally, we report Kodak PSNR and LPIPS so readers can see what human-facing quality the machine-oriented codec gives up.