Cross-modal hashing turns images and their captions into short bitstrings, typically 32 to 128 bits, so that a text query can find matching photos by counting differing bits instead of computing floating-point similarity. Practitioners often treat this compression as a privacy filter: surely 64 bits cannot hold a photograph. That same belief about continuous embeddings turned out to be wrong. Decoders trained to "translate" a vector back into its source recovered most short sentences verbatim, and turned image features into accurate captions. We propose to test the belief for hash codes directly, and then to build a private retrieval design around the answer. The design rests on a structural fact. When the server holds binary codes and the client's query is encrypted, the server can compute bit-difference counts using only additions and plaintext-by-ciphertext scaling. That means no encrypted multiplications, no noise-refresh steps, and no polynomial approximations. Encrypting the query therefore becomes cheap. What remains exposed is the database of stored codes, so the codes themselves must be made hard to invert.
The core claim has two parts, and each can be falsified. First, deep cross-modal hash codes at 64 and 128 bits leak substantially more content than their category labels alone would explain, because the training objectives used by recent methods pack as much meaning as possible into every bit. Second, a hash encoder trained against an inversion decoder can cut this excess leakage while keeping retrieval accuracy close to that of undefended hashing. Paired with homomorphically encrypted Hamming search, this gives cross-modal retrieval in which the query is cryptographically hidden and the store reveals little beyond coarse semantics. No differential-privacy noise is added to the query, and no continuous reranking stage is needed.
Where this sits
The closest work is Pointing the Way, Hiding the Destination. It runs text-only private retrieval in three stages. The client perturbs its query with directional differential-privacy noise (sampled from a von Mises–Fisher distribution, a Gaussian-like distribution on the sphere) before hashing. The server runs a plaintext Hamming filter over the corpus to produce a shortlist. The server then computes int8 dot products on the shortlist under BFV, a lattice-based homomorphic encryption scheme. That paper supplies our pipeline skeleton and serves as our main comparison. We differ in three ways. We work across modalities, retrieving images from text and the reverse. We replace query-side noise with encryption, since Hamming scoring is linear. And we target the threat it leaves untouched, which is leakage from the stored database codes.
The second closest is SCMR, which ran encrypted cross-modal retrieval using shallow matrix factorization, Paillier additive encryption, and locality-sensitive hashing. It predates generative inversion attacks and never asks whether its latent features leak content. We use deep hash encoders and make inversion the central measurement.
Two papers motivate the measurement. CapRecover turned continuous CLIP features into captions with ROUGE-L 0.53 on COCO2017 and recovered CIFAR-10 labels at 92.71% top-1. Its noise defense assumes a trusted party removes the noise before use, and it never examined binary codes. Transferable Embedding Inversion Attack showed that a few thousand leaked input-embedding pairs suffice to train a surrogate encoder and invert a victim's vectors without ever querying it. That is exactly the situation of a leaked hash store.
Eguard supplies the defensive method. It trains a projection to reduce mutual information with the source text, cutting reconstruction F1 to about 4–5% while SST-2 accuracy moved only from 94.3% to 93.8%. We adapt this idea to a binarized output trained jointly with a cross-modal objective, which is a harder setting because the code has very little capacity to spare.
TAPAS supplies the cryptographic intuition. It showed that binary values turn encrypted arithmetic into cheap XNOR and popcount operations, achieving a 29x MNIST speedup. It did this for classifiers, not retrieval.
Research Program
The work proceeds in three parts.
Part 1: measure leakage from hash codes. We take two recent hash encoders, AHLR and HINT, along with a simple baseline that binarizes CLIP features by sign. We train two kinds of attackers.
The first is a CapRecover-style decoder that maps the bitstring through a projection and a query-transformer module into a frozen language model. This attacker is trained on (image, code, caption) triples.
The second is a transfer attacker following Transferable Embedding Inversion Attack. It sees only 1k to 8k leaked (image or caption, code) pairs, fits a surrogate hash encoder, and inverts through the surrogate.
The key metric is excess leakage. Retrieval that works at the category level must reveal categories, so we do not count category recovery as a failure. Instead we compare the attacker's caption quality with a "label oracle" decoder that sees only the item's multi-label category vector. The difference is information the code leaks beyond what retrieval requires: counts, attributes, object relations, and text visible in the image. We also report label recovery and ROUGE-L for completeness.
Part 2: train inversion-hardened codes. We train the hash encoder with its usual cross-modal loss $\mathcal{L}_{ret}$ minus a penalty on a co-trained inversion decoder. The objective is
$$\min_{h}\ \mathcal{L}_{ret}(h) \;-\; \lambda \,\min_{g}\, \mathcal{L}_{cap}\big(g(h(x)) \mid c(x)\big),$$
where $h$ is the hash encoder, $g$ is the inversion decoder, and $\mathcal{L}_{cap}(\cdot \mid c(x))$ is the decoder's caption loss given the category labels $c(x)$. We implement the conditioning by feeding $c(x)$ to the decoder, so the penalty targets only excess information. The weight $\lambda$ trades accuracy against leakage. Binarization is relaxed with the usual straight-through estimator.
Adversarial training is known to fool its own adversary while leaving information intact. To guard against this, every reported leakage number comes from a fresh, larger attacker. It is trained from scratch after the encoder is frozen, with a decoder family different from the one used in training.
Part 3: encrypted Hamming search. Let $q \in \{0,1\}^b$ be the client's query code, sent encrypted, and $d \in \{0,1\}^b$ a stored code in plaintext. Their Hamming distance is
$$H(q,d) = \sum_{i=1}^{b} d_i + \sum_{i=1}^{b} q_i\,(1 - 2d_i).$$
This is linear in the encrypted bits $q_i$, with plaintext coefficients. The server can therefore score SIMD-packed database codes under BFV using only rotations, additions, and plaintext multiplications. A 64-bit code also occupies far fewer slots than a 768-dimensional int8 vector.
The server returns encrypted scores, or encrypted bucketed scores, and the client selects the top items. Payloads are fetched with oblivious transfer, a protocol that lets the client retrieve chosen items without the server learning which ones, as in Pointing the Way. We will measure latency, ciphertext communication, and evaluation-key size. As an ablation, we also encrypt the database codes, which makes $q_i d_i$ a ciphertext-by-ciphertext product of multiplicative depth one. This isolates the cost of protecting the store cryptographically rather than through learned codes.
One honest limitation: the client sees scores for every item it is sent, which is the score-leakage problem Pointing the Way also leaves open. We will quantify this leakage rather than claim to solve it.
Baselines
Undefended hashing (the naive reference). This is AHLR and HINT run as published, with codes stored in plaintext and queries sent in the clear. It sets the accuracy ceiling. AHLR reports image-to-text mAP 0.6797 at 64 bits on MS-COCO. HINT reports 75.5% image-to-text and 74.6% text-to-image MAP at 128 bits on MIRFlickr-25K. We will retrain both on identical splits, so our comparisons use our own reproduced numbers, with the published figures as sanity checks. This baseline also gives the leakage that Part 2 must reduce.
Cross-modal port of Pointing the Way (the strongest system competitor). We port Pointing the Way to CLIP-based image-text retrieval. It keeps von Mises–Fisher noise on the query before binarization, a plaintext Hamming shortlist of size K, and BFV int8 reranking. In text retrieval it retained 98.8%–100.2% of full-corpus NDCG@10 at K=500 and added 0.73 s on 2.68M-passage NQ. No cross-modal number for it was found on Lacuna.
It is the strongest competitor because it already balances accuracy, privacy, and speed in a deployed-style pipeline. We give it the same backbone, the same hash bit budget, and the same hyperparameter search budget over K and noise scale, and run it on the same hardware. Our design beats it if it matches its mAP within one point at equal or lower latency while also lowering store leakage. Its query is protected only statistically, and its store not at all.
Inversion defenses applied to continuous features before hashing. These are Eguard-style mutual-information projection and CapRecover's Gaussian obfuscation, applied to continuous features before a standard hash head. They show whether hardening must happen during binarization or can happen beforehand.
SCMR and the encrypted-store ablation. SCMR is the shallow cryptographic reference. Its paper reports under 1.5 ms secure search for a 500-dimensional vector, but no reported mAP on Wikipedia, NUS-WIDE, or IAPR-TC12 was found on Lacuna, so we rerun it.
Key-ingredient ablation. The ablation sets $\lambda=0$ while keeping the encrypted query. It isolates whether learned hardening, rather than encryption alone, is responsible for store privacy.
Why not just…?
Why not just trust binarization to be one-way? This is exactly what Part 1 tests, and it is a genuine possibility. If fresh attackers on 64-bit AHLR codes reach no more than the label oracle, hardening is unnecessary, and the contribution reduces to the cheap encrypted query. The evidence against assuming this is that continuous embeddings were wrongly assumed one-way (CapRecover). Moreover, contrastive hash training deliberately keeps relationships between items, which is what surrogate attacks exploit.
Why not just port Pointing the Way with directional noise? Its noise protects only the query, never the store. And AHLR found that cross-modal codes lose mAP when channels are perturbed, especially at 16–32 bits. That suggests the shortlist redundancy observed for text may not hold across modalities. The cross-modal port baseline measures this directly. If it holds mAP and its codes turn out not to leak, it wins, and we will report that.
Why not just apply Eguard to continuous features and hash afterward? This is a fair contender and is included as a baseline. Our concern is that the hash head is trained to recover as much meaning as possible from its input, which may reconcentrate the residual information. Whether it does is an empirical question that the before-versus-during comparison settles.
Why not just use two-server secure computation like p²RAG? p²RAG reports about 0.2 s and about 2 MB for 2^17 documents. It needs two non-colluding servers and a dealer, and it still exposes stored embeddings to whoever holds the database. We target a single server. We will report p²RAG's published figures alongside our communication measurements rather than claim parity.
Proposed Experiments
Retrieval and inversion experiments run on MS-COCO, MIRFlickr-25K, and NUS-WIDE at 32, 64, and 128 bits. The decisive check has three conditions. First, at 64 bits on COCO, undefended codes must show positive excess leakage over the label oracle. Second, hardened codes must cut that gap by at least half against fresh attackers. Third, hardened codes must lose at most two mAP points relative to the AHLR codes we reproduce.
Failure is informative either way. If undefended codes show no excess leakage, the "binarization is safe" view stands for cross-modal hashing. If hardening costs more than a few points of mAP, the trade-off is unfavorable at short code lengths.
The transfer attacker is swept over 1k to 8k leaked pairs to locate the leak size at which store protection collapses. Cryptographic cost is measured end to end on NUS-WIDE-scale corpora against the cross-modal Pointing-the-Way port. We vary BFV parameters chosen for the depth-zero query circuit, and compare them with the depth-one encrypted-store ablation.
References
- Pointing the Way, Hiding the Destination: Practical Private Dense Retrieval at Scale
- Enabling Secure Cross-Modal Retrieval Over Encrypted Heterogeneous IoT Databases With Collective Matrix Factorization
- CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models
- Transferable Embedding Inversion Attack: Uncovering Privacy Risks in Text Embeddings without Model Queries
- Mitigating Privacy Risks in LLM Embeddings from Embedding Inversion
- TAPAS: Tricks to Accelerate (encrypted) Prediction As a Service
- Dynamic Masking and Auxiliary Hash Learning for Enhanced Cross-Modal Retrieval
- Hierarchical Encoding Tree with Modality Mixup for Cross-modal Hashing
- $p^2$RAG: Privacy-Preserving RAG Service Supporting Arbitrary Top-$k$ Retrieval