Search systems learn best from their own query logs, and those logs are among the most sensitive data a company or hospital holds. Differential privacy (a formal guarantee that the trained model would look almost the same whether or not any single user's data was included) offers a principled way to train on them. Its standard training algorithm, DP-SGD, clips each example's gradient to a fixed size and adds noise. The usual recipe for training neural retrievers is contrastive learning with in-batch negatives: each query is scored against its matching passage and against every other passage in the batch. That recipe breaks DP-SGD in two ways. First, each example's loss depends on other users' passages, so the influence of one user is no longer confined to one clipped gradient. Second, contrastive training wants very large batches, which is exactly where per-example gradient memory runs out.
We propose a simple structural fix and argue that it also makes hybrid retrieval privatizable end to end. The document encoder is frozen after public pretraining, and only the query encoder is trained privately against a fixed index of public-model document vectors. Under this choice each example's loss touches only that example's query text, so per-example clipping is well defined. Ghost clipping, a trick that computes each example's gradient size without building the gradient itself, then applies directly, and negatives from the whole index come at almost no memory cost. Because the sparse, keyword side of the hybrid system is either untrained (BM25 over the corpus) or derived from the privately trained query encoder, the whole hybrid retriever inherits one privacy budget. Our central prediction is that privacy noise mostly damages queries that hinge on rare exact strings, which is what the keyword path is good at. If so, the hybrid system should lose far less under privacy than a dense-only system.
Core Claim
The claim has three linked parts. The first is a design claim: freezing the document tower and training only the query tower privately is the natural unit of privacy for retrieval trained on query logs. It gives a correct sensitivity bound, and it lets ghost clipping scale contrastive DP training to index-sized negative pools. The second is an empirical claim: at a fixed privacy budget, most of the drop from non-private to private dense retrieval sits on queries with rare entities or identifiers, and fusing with a lexical path recovers most of it. The third is a methodological claim about text preprocessing. For a sequence input, a linear layer's per-example gradient is a sum of per-token outer products, and its squared norm is
$$\|\nabla_W L_i\|_F^2 = \sum_{t,s} (g_{i,t}^\top g_{i,s})(x_{i,t}^\top x_{i,s}),$$
where $x_{i,t}$ is the input activation and $g_{i,t}$ the backpropagated error at token $t$ of example $i$. This norm grows with the number of tokens that carry signal. Long, boilerplate-heavy queries, such as conversational histories full of filler, are therefore clipped harder. Preprocessing that removes this material acts as a sensitivity control, not just a cleanup step. Because it is applied per record before training, it costs no privacy budget.
This differs from the closest work. Large Language Models Can Be Strong Differentially Private Learners introduced ghost clipping for Transformer fine-tuning on classification and generation. Its losses are per-example by construction, it never faces the cross-example coupling of contrastive retrieval, and it keeps the input pipeline fixed. The hybrid retrievers of Sparse, Dense, and Attentional Representations for Text Retrieval, Contextualized Query Embeddings for Conversational Search and A Proposed Conceptual Framework for a Representational Approach to Information Retrieval are all trained without privacy. The file contains no DP-trained retriever at all.
Research Program
We would build three components.
The private query tower. A BERT-sized bi-encoder is pretrained non-privately on public data. The document side is frozen, and every corpus passage is encoded once into an approximate nearest-neighbor index. Private training then minimizes
$$L_i = -\log \frac{\exp(\eta_q(q_i)^\top e_{d_i^+})}{\sum_{d \in N_i \cup \{d_i^+\}} \exp(\eta_q(q_i)^\top e_d)},$$
where $\eta_q$ is the trainable query encoder, $e_d$ is the frozen vector for passage $d$, $d_i^+$ is the passage the user clicked, and $N_i$ is a set of negatives drawn from the public index. The negatives must not come from other private examples. They may be chosen by the public model, or by a retrieval step using the current private model, which is post-processing of a DP model and so safe.
Since $e_d$ is a constant, the gradient of $L_i$ flows only through $q_i$'s tokens. The per-example norm is exactly the ghost-clipping quantity above. The softmax over thousands of negatives adds one matrix product and no per-example parameter memory. We would implement this on top of the ghost-clipping formulation of Li et al. and report memory and throughput against explicit per-example gradients. The two-pass backward cost noted for ghost clipping remains, but it now applies only to queries, which are short.
The hybrid. We would compare two ways of adding a lexical path. The first is plain BM25 over the corpus, fused with the private dense score by linear interpolation as in Sparse, Dense, and Attentional Representations for Text Retrieval. The second is a CQE-style path from Contextualized Query Embeddings for Conversational Search. CQE weights query terms by the norms of their contextual token embeddings and sends those weighted terms to BM25. Here the embeddings come from the privately trained query encoder, so the term weights are post-processing and add no privacy cost.
We would state the threat model explicitly. The protected records are query-log entries. The corpus is treated as the searchable collection, not as secret training data. This is exactly the leakage question raised by the asymmetric-privacy gap: a sparse index mined from private training text would not be covered by the guarantee, and our design deliberately avoids building one.
The preprocessing study. We would compare three query pipelines on a fixed privacy budget. The first is raw text. The second is conventional normalization: stop-word removal, punctuation stripping and lemmatization, as used by Machine learning for pattern detection in cochlear implant FDA adverse event reports. The third is an entity-preserving compression that strips filler and markup but never touches alphanumeric tokens, codes or rare capitalized strings. For conversational queries, this third pipeline also shortens the history. It is inspired by the fluff-stripping bottleneck of Benchmarking LLMs' Judgments with No Gold Standard, but implemented with rules rather than an LLM so that it stays a per-record transformation.
Two lines of prior work constrain this design. Crowson et al. also observed that bag-of-words cleaning loses clinical qualifiers. The general concern that aggressive cleaning erases identifiers that sparse indexes need is exactly the damage the private hybrid depends on avoiding. We would measure the effect of each pipeline on three things: the distribution of per-example gradient norms, the fraction of examples clipped, and retrieval quality.
Baselines
Zero-shot public model (do nothing). The naive reference point is to never touch private data: take the publicly pretrained bi-encoder, fused with BM25, and deploy it as is. It is the real competitor, because a private model that cannot beat it has no reason to exist. It gets the same architecture, index and fusion weight as our system; the fusion weight is tuned on a public development split. Private training beats it if, at ε = 3 or lower, it improves MRR@10 or nDCG@3 on the private domain by a margin larger than seed-to-seed variation.
Non-private training (upper reference). Here the same query-tower-only setup is trained without clipping or noise. This isolates the cost of privacy from the cost of freezing the document tower. We would also report the published non-private MS MARCO numbers for context. Sparse, Dense, and Attentional Representations for Text Retrieval reports MRR@10 of 0.302 for single-vector DE-BERT and 0.343 for ME-HYBRID. A Proposed Conceptual Framework for a Representational Approach to Information Retrieval reports roughly 0.378 for TCT-ColBERT fused with DeepImpact. On conversational search, Contextualized Query Embeddings for Conversational Search reports nDCG@3 of 0.515 for CQE-hybrid on TREC CAsT. There is no reported result on Lacuna for any differentially private retriever. Our DP numbers would therefore be judged against the zero-shot floor and our own non-private ceiling, not against a prior private system.
Two-tower private training (strongest existing method). The closest existing approach is to apply the Li et al. recipe directly: both towers trainable, in-batch negatives, per-example clipping. To keep it honest, we would give it a correct sensitivity accounting. Each private pair is treated as one example, and negatives are drawn only from public passages, so its guarantee matches ours. Both methods get the same GPU memory, the same number of optimizer steps, and the same hyperparameter search over learning rate, clip norm and batch size, since DP-Adam is known to be sensitive to these. Beating it means equal or better retrieval at equal ε with less memory, or clearly better retrieval at equal memory, because the frozen tower allows larger negative pools.
Ablations isolating the key ingredients. The first ablation fuses the private dense tower with plain BM25 instead of the CQE-style private term weights. This isolates whether learned sparse weights matter. The second varies the size of the negative pool, which isolates the benefit that the frozen tower makes affordable. The third swaps preprocessing pipelines while holding everything else fixed.
Why not just…?
Why not just use BM25 with good preprocessing and skip neural training and privacy entirely? BM25 needs no training, so it leaks nothing about query logs. It is also the system that supervised models fail to beat out of domain, according to A Proposed Conceptual Framework for a Representational Approach to Information Retrieval. Its weaknesses are vocabulary mismatch and multi-turn queries full of pronouns, and those are where CQE-style hybrids gain. BM25 is reported in every table. If private hybrid training fails to beat it on the conversational benchmark, we would conclude that at realistic budgets privacy erases the learned component's value.
Why not just DP-train the dense tower and fuse it with ordinary BM25? This is a real threat to the learned-sparse part of the proposal, and it may well work as well. It is precisely the first ablation. If plain BM25 fusion matches the CQE-style private term weights, we would report that and drop the learned sparse path. The core claims do not depend on it: correct sensitivity through the frozen query tower, and hybrid fusion concentrating its recovery on rare-entity queries.
Why not just train both towers with Opacus and in-batch negatives? First, in-batch negatives make each loss depend on other users' passages, so a naive per-example clip understates sensitivity. Second, materializing per-example gradients limits batch size. For comparison, Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy reports 18x larger maximum batches than Opacus with ghost-style clipping on VGG19, and Li et al. report about 3x lower memory. The two-tower baseline with corrected accounting tests whether training the document tower is worth the cost. If it wins clearly, the frozen-tower claim fails.
Why not just scrub identifiers from the logs and train without DP? Scrubbing gives no formal bound. Crowson et al. found that stylistic "reporting signatures" alone let a linear model recover the manufacturer of cochlear implant adverse-event reports with 88.6% accuracy after standard cleaning, so records stay identifiable without names. We would also run a canary audit on a scrubbed, non-private model, described below, to show whether memorized strings survive.
Proposed Experiments
Data and training. The public stage pretrains on Natural Questions. The first private stage treats MS MARCO passage training queries as a simulated private log and evaluates with MRR@10 on the dev set. The second private stage treats CANARD-derived conversational training data as private and evaluates with nDCG@3 on TREC CAsT-19 and CAsT-20. Budgets are ε ∈ {1, 3, 8} with δ below the inverse of the log size.
Test 1: where does privacy hurt? We split evaluation queries by whether the relevant passage shares a rare token (low document frequency, or alphanumeric) with the query. We then measure the DP-minus-non-DP gap for the dense-only and hybrid systems in each slice. The prediction is that the dense gap is concentrated in the rare-token slice and that the hybrid closes most of it. If instead the damage is uniform across slices, the rare-entity story is wrong.
Test 2: does preprocessing reduce clipping? We plot per-example gradient norm against query length under the three pipelines, together with the clipped fraction and retrieval quality. The prediction is that entity-preserving compression shrinks the norm spread and improves retrieval at fixed ε. Conventional lemmatization should hurt the rare-token slice even when it reduces clipping.
Test 3: is the privacy empirical as well as formal? We insert synthetic canary queries containing fake identifiers into the private log. We then test whether each trained model ranks the canary's paired passage abnormally high for partial prompts, and compare the DP model, the scrubbed non-DP model and the raw non-DP model.
Test 4: memory and throughput. We measure memory and throughput of ghost clipping on the query tower as the negative pool grows from hundreds to tens of thousands, against explicit per-example gradients.
What would sink the project. The whole program rests on one comparison: private hybrid training at ε = 3 must beat the zero-shot public hybrid on CAsT. If it does not, the program is not worth pursuing.
References
- Large Language Models Can Be Strong Differentially Private Learners
- Sparse, Dense, and Attentional Representations for Text Retrieval
- Contextualized Query Embeddings for Conversational Search
- A proposed conceptual framework for a representational approach to information retrieval
- Machine learning for pattern detection in cochlear implant FDA adverse event reports
- Benchmarking LLMs' Judgments with No Gold Standard
- Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy