Large retrieval systems store each item, for example a photo together with its caption or tags, as a short binary code. Two codes can then be compared with a single XOR-and-count instruction, which is what makes search over millions of items fast. The cheapest way to train such codes is point-wise supervision. Instead of comparing every pair of items, a cost that grows with the square of the dataset, each item is pulled toward one fixed "hash center," a target binary code assigned to its category. Today those centers are drawn at random or from a Hadamard matrix, so that every category sits at the same distance from every other. As a result "Labrador" is exactly as far from "Golden Retriever" as from "Toaster." Multi-label items get an awkward average of several centers, and that average no longer lies on a valid binary code.
We propose to write the centers in language instead. Each item's label set becomes a short phrase such as "dog, grass, frisbee." The same text encoder that reads the item's own caption embeds that phrase. A small learned projector, trained to keep centers a guaranteed minimum distance apart, then maps the embedding to a binary center. Items are encoded by fusing their image and text features early, before any code is produced, and each item is trained against the center of its own label set. Training stays linear in dataset size. Centers now inherit the semantic geometry of language, and every label combination gets its own exact binary target rather than an average of several.
Core Claim
In fused image–text hashing, the main loss of retrieval quality comes from where the targets are placed, not from the feature extractor. Moving to centers projected from language and compositional over label sets should give gains that swapping in a stronger backbone alone does not. The gains should be largest in three places: multi-label data, long codes, and queries where one modality is missing.
The closest prior system is Central Similarity Multi-View Hashing (CSMVH). It fuses ResNet-50 and BERT features with a gated multimodal unit, which is a learned sigmoid gate weighting each modality per input, and trains against data-independent Hadamard or Bernoulli centers. Its authors report that accuracy plateaus from 64 to 128 bits, and it leaves multi-label centers unresolved. CLIP Multi-modal Hashing (CLIPMH) supplies the pre-aligned CLIP backbone and context-gated fusion, but it trains with a pairwise metric loss and has no centers at all. Deep Hashing with Semantic Hash Centers (SHC) motivates our work directly: it shows that data-dependent centers beat CSQ, MDS and OrthoHash by 11.71% MAP@ALL on fine-grained image benchmarks. SHC derives its centers from a ResNet34 confusion matrix, which only works for single-label images and requires solving an NP-hard assignment problem once per label vocabulary. Deep Cross-modal Proxy Hashing (DCHML) learns category proxies and handles multi-label items by averaging them. Its authors name BERT-initialized proxies as future work, and it never fuses the two modalities. Our proposal is the combination none of these makes: centers generated through a cross-modal text projection, compositional over label sets, supervising an early-fused encoder.
Research Program
The item encoder takes frozen CLIP image and text embeddings $x_v$ and $x_t$. A gated fusion unit combines them into a single vector, and a small MLP followed by a tanh maps that vector to a relaxed code $h \in (-1,1)^K$, where $K$ is the code length in bits. The center generator encodes the label-set phrase $s$ with the frozen CLIP text encoder $\phi$. A learned linear-plus-MLP projector $P$ maps the result into $K$ dimensions, giving the center $c_s = \mathrm{sign}(P(\phi(s)))$. Because $\phi$ also reads item captions, the centers live in the same geometry the fused features come from. This is the sense in which the proposal uses cross-modal projection: text becomes the coordinate system for the binary targets, rather than serving only as an input feature.
The projector is trained first, on label sets alone, with this objective:
$$ \mathcal{L}_P = \sum_{s,s'} \Big( \tfrac{1}{K}\, \tilde c_s^\top \tilde c_{s'} - \cos(\phi(s),\phi(s')) \Big)^2 + \lambda \sum_{s \neq s'} \max\!\big(0,\ d_{\min} - d_H(\tilde c_s, \tilde c_{s'})\big). $$
Here $\tilde c_s = \tanh(P(\phi(s)))$ is the relaxed center, and $d_H$ is the relaxed Hamming distance $\tfrac{K}{2}(1 - \tfrac{1}{K}\tilde c_s^\top \tilde c_{s'})$. The margin $d_{\min}$ is set from the Gilbert–Varshamov bound, as in SHC. The first term makes Hamming similarity track language similarity. The second term forbids collisions and so directly addresses crowding when there are many classes. This optimization runs over the set of distinct label sets, which on NUS-WIDE and COCO is far smaller than the number of items. It is therefore cheap and has no dependence on the number of items $N$.
The item encoder is then trained against the frozen centers. The loss is CSQ's central-similarity cross-entropy between $h$ and $c_{s(i)}$, plus a log-cosh quantization penalty, following Central Similarity Quantization. Each sample has exactly one target, so training cost is $O(N)$. Label sets that never appear in training get a center at test time by projecting their phrase. This lets us test how compositional the centers really are.
We will build three things for analysis. First, a measurement of how far each center scheme distorts the label-similarity structure: rank correlation between center Hamming distances and label Jaccard similarity. Second, per-bit usage statistics, to test whether semantic centers use the extra bits that CSMVH leaves idle at 128. Third, a missing-modality variant, in which the gate is trained with random modality dropout so that image-only or text-only queries can search a fused database.
Baselines
The naive reference point is untrained binarization. We concatenate the CLIP image and text embeddings, apply random-hyperplane locality-sensitive hashing (random projections followed by sign), and retrieve by Hamming distance. This shows how much any supervised hashing adds on top of CLIP.
The strongest existing competitors are CSMVH and CLIPMH. We will not rely on their original backbones. We will re-implement both on the same frozen CLIP ViT features, the same fusion unit, the same splits, and the same code lengths of 16, 32, 64 and 128 bits. Each method gets the same hyperparameter search budget and the same number of training epochs; CSMVH reports stabilizing within about 40 epochs. CLIP-backed CSMVH is the exact "obvious idea" at this intersection, so it is the baseline that matters most. The file reports only relative gains for these systems: CSMVH gains up to 11.41% mAP over BSTH on MS COCO, and CLIPMH gains 6.82% on COCO, 2.51% on MIR-Flickr25K and 1.83% on NUS-WIDE over DMMVH. No reported absolute multi-view mAP for either system was found on Lacuna. We will therefore report our re-run numbers alongside their published protocol. Beating these baselines means a consistent mAP gain over CLIP-backed CSMVH at every bit length on both multi-label benchmarks, larger than seed-to-seed variance across five seeds, together with training time no worse than CSMVH's.
The ablations each swap out one ingredient while keeping everything else fixed. For the target, we compare our centers with Hadamard centers, with DCHML-style averaged per-label centers, and with SHC-style confusion-derived centers where labels permit. For the projector, we compare the learned $P$ with a fixed random projection of $\phi(s)$ that is repaired to the minimum distance afterwards. For supervision, we compare per-label-set phrases with per-label centers that are then combined. For fusion, we compare gated early fusion with late fusion that averages separate image and text codes.
Why not just…?
Why not just give CSMVH a CLIP backbone? This is the most serious threat. CLIPMH's gains suggest that much of the recent progress comes from the backbone. Our main comparison holds features and fusion fixed and changes only the targets, so this experiment settles the question directly. If CLIP-CSMVH matches us on single-label-like splits but falls behind on items with three or more labels and at 128 bits, the claim survives in narrowed form. If it matches us everywhere, the claim is wrong.
Why not just use SHC's semantic centers? SHC builds similarity from a classifier's confusion between classes. That quantity is undefined for label combinations, and it requires training an extra classifier. We will run SHC on per-label centers averaged in the DCHML manner. The prediction is that it trails us specifically on rare label sets and does not generalize to held-out ones.
Why not just sign-binarize the label text embeddings with random hyperplanes? This is a real possibility. CLIP text geometry might already be good enough that the learned projector adds nothing. The fixed-random-projection ablation tests this. If it matches us, we will report the simpler method as the contribution and drop the learned $P$. The minimum-distance repair would stay, since without it the random codes may collide when there are many classes.
Proposed Experiments
The main experiments cover multi-view retrieval, where both queries and database items are image-plus-text, on MIR-Flickr25K, NUS-WIDE and MS COCO. We use the CSMVH and CLIPMH protocols and report mAP at 16 to 128 bits, precision–recall curves, and training wall-clock time. IAPR TC-12 serves as a fourth multi-label set.
The first decisive check stratifies mAP by the number of labels per query. The core claim predicts that our margin over averaged or Hadamard centers grows with the number of labels.
The second decisive check is the 64-to-128-bit slope, where CSMVH reported a plateau. We predict that semantic centers keep improving, and the bit-usage analysis should show why.
The third check is compositional generalization. We hold out 10% of label sets during training, obtain their centers purely by projection at test time, and measure retrieval of those items against items whose centers were seen in training.
As a secondary track, we run image-only and text-only queries against a fused database using the modality-dropout variant. These are compared with DCHML-style separate encoders. For reference, DCHML reports 0.790 mAP on NUS-WIDE text-to-image at 128 bits, but under its own features and splits, so we treat that number as context rather than a direct target.
All systems use the same frozen CLIP features, so any difference we observe can be attributed to where the targets sit and how the modalities are fused.