Why Short Binary Codes Fail on Frozen Foundation Models in Shifted Domains

Large image search systems compress each item into a short binary code, say 16 to 64 bits, so that comparing a query against millions of items is just counting differing bits. A common recipe today is to freeze a large pre-trained model such as CLIP or DINOv2 and train only a small "hash head" that turns its final embedding into bits. On web photos this works well. Work on adapting frozen models suggests it should break on specialized data such as pathology slides or night-time street scenes, where the frozen model never learned the right features. Nobody has measured whether turning features into bits makes that failure worse, or why.

This proposal splits the accuracy lost by short codes into two parts. The first is rigidity: the frozen final embedding never contained the needed information. The second is quantization: the information is present but is destroyed when squeezed into a few bits. We test a concrete account of where the second loss comes from in shifted domains. The class-relevant signal sits in low-variance directions of the frozen embedding, and partly in intermediate layers the final embedding has discarded. Few-bit codes are the first place that signal gets thrown away. If the account holds, the fix is a small side network that reads intermediate frozen layers and is trained end-to-end through a gradually sharpened sign function, without backpropagating into the backbone. The decomposition then tells a practitioner, before training anything expensive, whether a frozen head is enough.

Core Claim

We make three linked claims. First, the gap between hashing frozen features and hashing adapted features is not constant: it grows as the bit budget shrinks and as the domain moves away from pre-training data. Second, much of the low-bit loss in shifted domains comes from discriminative information in low-variance or intermediate-layer directions. A projection that favours high-variance directions, or that sees only the final layer, cannot keep it in $q$ bits. Third, a detached side network over intermediate layers recovers most of the gap to full fine-tuning at a fraction of the memory, but only where the first two claims predict a gap exists.

The spectral part of the hypothesis borrows from compression research. Restoring Pruned Large Language Models via Lost Component Compensation restores pruned language models by adding back activation directions lost in pruning. It found that minor SVD components (directions with small singular values) often carry the signal needed for hard reasoning. We ask whether the same is true of what a frozen backbone needs for fine-grained retrieval in a new domain. Short codes are exactly the regime that discards minor directions first.

Relation to Prior Work

Three papers motivate the gap from the hashing side. Simultaneous feature learning and hash coding with deep neural networks trained the feature extractor jointly with the code and raised SVHN mAP from about 0.58 (hand-crafted features with a separate hashing step) to over 0.89. HashNet: Deep Learning to Hash by Continuation replaced the sign function with $\tanh(\beta z)$ and raised $\beta$ during training so codes become exactly binary. Both updated entire small convolutional networks, which is impractical for billion-parameter backbones. CLIP Multi-modal Hashing for Multimedia Retrieval takes the opposite position: it freezes CLIP, trains only a gating-and-hash head, and reports mAP gains on MIR-Flickr25K, NUS-WIDE and MS COCO. This seems to contradict the joint-training thesis. Our decomposition predicts both are right in their own regimes: frozen heads suffice on web-like data at moderate bit lengths, and fail in shifted domains at low bit lengths. We test that prediction rather than assume either side.

Two papers supply the method. Res-Tuning attaches tuners as parallel branches whose gradients never enter the backbone, cutting training memory substantially. It was only evaluated on continuous classification and generation. SelaVPR++ is the closest existing work. It uses side adapters on DINOv2 with binary codes trained by a straight-through estimator, which passes gradients around the sign function as if it were the identity. It relies on a floating-point re-ranking stage to rescue binary-stage errors, and notes that hard cases currently require larger descriptors.

We differ from SelaVPR++ in three ways. We study single-stage binary retrieval and use re-ranking only as a comparison. We compare continuation against straight-through training inside the side network. Most importantly, our contribution is the measured decomposition and the spectral mechanism, not another adapter.

Two papers constrain the design. Bridging the Pathology Domain Gap shows full fine-tuning of CLIP on small pathology sets overfits and forgets. Few-shot Adaptation of Medical Vision-Language Models shows a linear probe often beats prompt learning when the backbone lacks domain knowledge. Together they say heavy adaptation is not automatically better. Our method therefore has to beat simple frozen heads, not just full fine-tuning.

Research Program

The first component is a measurement. For a backbone, dataset and bit length $q$, define four quantities. $m_{\text{ft}}$ is the retrieval mAP of continuous features from a fully fine-tuned backbone, the reference ceiling. $m_{\text{cont}}$ is the mAP of continuous frozen final-layer features, with a learned supervised projection but no binarization. $m_{\text{head}}(q)$ is the best frozen-final-layer binary head at $q$ bits. $m_{\text{side}}(q)$ is our side network at $q$ bits. The decomposition is

$$ m_{\text{ft}} - m_{\text{head}}(q) = \underbrace{(m_{\text{ft}} - m_{\text{cont}})}_{\text{rigidity}} + \underbrace{(m_{\text{cont}} - m_{\text{head}}(q))}_{\text{quantization}}, $$

and we track how each term moves with $q$ and with domain shift.

To probe the mechanism, we fit a supervised linear probe on frozen features. We then measure what fraction of its weight energy falls in the bottom half of the feature covariance's principal components. We call this the minor-direction share, and repeat it per layer. The prediction is that the minor-direction share is higher in shifted domains, and that it correlates across datasets with the quantization gap at 16 and 32 bits.

The second component is the side network. A small convolution-plus-MLP branch reads detached features from several intermediate blocks of the frozen backbone plus the final embedding. It outputs $q$ values passed through $\tanh(\beta z)$, with $\beta$ increased on HashNet's schedule. It is trained with a pairwise similarity loss weighted for the positive/negative pair imbalance HashNet describes, plus a quantization penalty. Only the branch receives gradients, so memory stays close to inference cost.

Baselines

The naive reference is frozen final-layer features compressed without supervision, using PCA followed by an optimized rotation and sign (iterative quantization). This is the decoupled pipeline the hashing literature argues against, and it is the purest test of the variance hypothesis because it keeps high-variance directions by construction.

The strongest frozen competitor is a CLIPMH-style supervised head: gating plus an MLP hash layer on frozen CLIP features, trained with metric and quantization losses as in CLIP Multi-modal Hashing for Multimedia Retrieval. That paper reports only relative gains over DMMVH, for example +6.82% mAP on MS COCO, so we reimplement it. A supervised head is not variance-driven, so it can in principle find minor directions. It is therefore the baseline the side network must beat. Beating it means a gain of at least 3 mAP points at 16 and 32 bits on the shifted-domain datasets, averaged over three seeds with non-overlapping standard deviations. On web datasets we expect parity and would report that as confirmation, not failure.

The adapted competitors are in-backbone LoRA trained with the same hashing loss, and full fine-tuning with HashNet's loss. The latter is the cost-unconstrained ceiling. The only absolute number on the file for this style of method is HashNet's 16-bit ImageNet mAP of 0.5059 in HashNet, obtained with an older convolutional backbone. We use it only as a sanity anchor when reproducing HashNet's loss.

For place recognition the strongest system is SelaVPR++, run both with and without its float re-ranking stage. The file gives no reported recall numbers for it on Tokyo24/7, Pitts30k or Nordland, so we rerun it.

Fairness rules apply to every method. All share the same backbone (CLIP ViT-L and DINOv2-large), the same training splits, the same number of hyperparameter trials (twenty per method per dataset), and the same epoch budget. We report trainable parameters and peak training memory alongside mAP.

Two ablations isolate the key ingredients. The side network restricted to the final layer separates "reading intermediate layers" from "a nonlinear head". Swapping continuation for straight-through estimation separates the training rule from the architecture.

Why not just…?

Why not just train a better hash head on frozen final features? This is the real threat. If a supervised head already extracts the minor directions, the side network adds nothing. The final-layer-only ablation and the head baseline settle it directly. If $m_{\text{head}}(q)$ matches $m_{\text{side}}(q)$ on PCam and NCT-CRC at 16 bits, we conclude the low-bit loss is a head-training problem, not missing information, and report that frozen heads suffice. That would be a useful, publishable answer. The rigidity term $m_{\text{ft}} - m_{\text{cont}}$ tells us beforehand how much room any final-layer method has.

Why not just fine-tune the backbone with LoRA or fully? Path-CLIP reports that full fine-tuning on scarce pathology data overfits and forgets. Its adapter beats standard CLIP adapters by 19% accuracy at 0.1% labels on PCam. Res-Tuning and SelaVPR++ document the memory cost of backpropagating through large backbones. We still run both, because on larger datasets such as MSLS full fine-tuning may win outright. The claim we defend is accuracy per unit of training memory, with full fine-tuning as the ceiling.

Why not just use more bits, or re-rank with float vectors? More bits is the trivial fix SelaVPR++ itself points to. We therefore report full mAP-versus-bits curves and ask whether the side network at $q$ bits matches the head at $2q$ bits. That comparison gives the gain a concrete storage value. Re-ranking cannot recover a true match the binary stage never retrieved, so we measure candidate recall at the top 100 under both codes.

Why not just heal the frozen model with soft prompts, as in compress-then-prompt? Compress, Then Prompt shows learned input prompts can repair a weight-compressed language model, lowering 2-bit OPT-1.3B perplexity from 2337.8 to 59. Prompts must backpropagate through the whole backbone, and the medical few-shot study found CoOp-style prompting used 28GB versus 800MB for a linear probe, while still underperforming it. We include visual prompt tuning with the hashing loss as one extra baseline on the medical datasets only, to check this directly.

Proposed Experiments

Datasets span a shift axis. Web-like retrieval uses CIFAR-10, NUS-WIDE and MS COCO. Shifted retrieval uses PCam, MHIST and NCT-CRC-HE-100K, with label-defined relevance for mAP. Condition-shifted place recognition uses MSLS, Tokyo24/7 and Nordland, scored by recall at 1. Codes are 16, 32, 64 and 128 bits. For the medical sets we found no reported hashing result on Lacuna, so every number there will be new.

The first decisive check is the interaction. The frozen-head-versus-side gap should be near zero on CIFAR-10 at 64 bits and largest on PCam or Nordland at 16 bits. If the gap is flat across that grid, the central claim fails.

The second decisive check is the mechanism. Across the nine datasets and two backbones, the minor-direction share should rank-correlate positively with the quantization gap at 16 bits. The share measured in intermediate layers should predict the side network's gain over the final-layer ablation.

The third check is cost. The side network should reach at least 80% of the distance from head to full fine-tuning while using a small fraction of full fine-tuning's training memory, in line with the memory savings SelaVPR++ and Res-Tuning report for detached adapters.

Negative outcomes on any of these checks are informative and will be reported as such.