Picking Training Data Mixtures for an Encrypted Target in One Round

A model developer often wants to tune a language model for a client whose data it is not allowed to see, such as a hospital, a law firm, or a company's internal codebase. The first decision is a data mixture: what fraction of training tokens to draw from each of the developer's sources (web text, code, papers, dialogue) so the model fits the client's distribution. Today's best mixture optimizers need the target data in the clear. Encrypting it with homomorphic encryption (a cryptosystem that lets a server add and multiply numbers it cannot read) is known to make neural computation 4–5 orders of magnitude slower. So running these optimizers "under encryption" looks hopeless.

We argue the hopeless part can be skipped. A mixture is a vector of perhaps 7–22 numbers. To choose it, the developer does not need a neural model run on the client's text. It needs the client's aggregate token-pattern counts, combined with per-source statistics the developer already holds, through a computation that is linear in the encrypted data. That computation fits in a single encrypted inner product per candidate mixture, with no bootstrapping and no encrypted nonlinearities. The open question is empirical: whether mixtures chosen this cheaply transfer to neural models nearly as well as mixtures from neural-proxy methods that see the target in plaintext. The project is built to answer that question and to map the cost-versus-quality frontier if the answer is "partly."

Core Claim

Suppose each developer source $k$ is summarized by a smoothed, hashed n-gram distribution $\theta_k$ over $V$ hashed buckets. The client summarizes its private target set by the count vector $C$ of those same buckets. Then the target log-likelihood of the token-level mixture with weights $\lambda$ is

$$ f(\lambda) = \sum_{v=1}^{V} C_v \log \Big( \sum_{k=1}^{K} \lambda_k \theta_{k,v} \Big), $$

where $C_v$ is the client's count in bucket $v$, $\theta_{k,v}$ is source $k$'s probability for that bucket, and $\lambda$ lies on the probability simplex.

For any candidate $\lambda$, the vector $\log(\Theta\lambda)$ is plaintext that the developer can compute itself. The score $f(\lambda)$ is therefore an inner product between an encrypted vector and a plaintext vector. In the CKKS scheme (an approximate-arithmetic homomorphic scheme), that costs one plaintext multiplication and a logarithmic number of slot rotations. Using the SIMD batching that CryptoNets introduced (packing thousands of values into one ciphertext), hundreds of candidates can be scored together. Only the scores, or a differentially private argmax over them, are ever decrypted. The client never reveals its counts, and the developer never reveals its source tables.

Because $f$ is concave in $\lambda$, this is the same kind of convex mixture problem that MixMin derives. MixMin trains one proxy model per source and optimizes a mixture of their output distributions by entropic descent against target samples. It reports a 1–5% relative improvement in negative log-likelihood at under 0.2% extra compute.

The constraint comes from MixMin itself. Its convex reduction is justified as proxy expressivity grows, and n-gram proxies sit at the opposite end. Our bet is that the mixture decision is low-dimensional and insensitive enough for weak but exactly encryptable proxies to recover most of the gain. Supporting evidence comes from Magnusson et al., who find that small-scale likelihood-based rankings predict large-scale data choices about 80% of the time, and from RegMix, which ranks mixtures with Spearman 0.97–0.98 using 1M-parameter proxies. If the bet fails, the project measures how far up the expressivity ladder one must climb, and at what encrypted cost.

Relation to Prior Work

The closest work is DUET. It also targets clients with end-to-end encrypted evaluation data. It wraps Bayesian optimization around LoRA fine-tuning runs and receives only noisy scalar feedback from the client, needing about 10 iterations (around 10 GPU-hours) to reach a near-optimal mixture. DUET treats encryption as an administrative wall: every query costs a training run, and the authors note that the loop is prohibitive at pre-training scale. Our proposal differs in two ways. It computes over the ciphertext directly. It also needs zero training runs and one communication round before the mixture is fixed.

Two other papers motivate the setting but stop short of mixture weights. FHAIM runs a full synthetic-tabular-data pipeline under CKKS. Its useful lesson for us is to replace non-polynomial scores with low-degree ones and to decrypt only differentially private selections. We reuse that design pattern, but apply it to text mixture weights rather than tabular marginals. Continual Pretraining on Encrypted Synthetic Data states that homomorphic encryption is too heavy for pre-training and uses deterministic AES on entities instead. We agree about training. Our point is that the mixture decision is far cheaper than training.

The overhead literature constrains which designs are allowed. Encrypted SGD in Towards Deep Neural Network Training on Encrypted Data took about 40 minutes per 60-sample mini-batch on a tiny MLP. Polynomial activation approximations trade accuracy against multiplicative depth (Chen et al.). Bootstrapping, the costly procedure that refreshes noisy ciphertexts, dominates cost (Circuit Bootstrapping). And ciphertext plus key material can reach tens of gigabytes per query (Haralampieva). Together these rule out running MixMin's neural proxies over encrypted target text as a default. That design survives only as the expensive top rung of our ladder.

Research Program

We build three tiers of encrypted mixture scoring and measure each one's encrypted cost and the quality of the mixture it picks.

Tier 1: hashed n-gram counts. This is the core claim above. The client hashes unigrams and bigrams into $V$ buckets, with $V$ swept from $2^{14}$ to $2^{20}$, and encrypts $C$. The developer packs $\log(\Theta\lambda)$ for $M$ candidate mixtures and returns encrypted scores. Candidates come either from a simplex grid or from a short schedule: the developer proposes a batch, the client returns a noisy argmax, and the developer refines around it. Multiplicative depth is 1. Cost scales with $V M / \text{slots}$ ciphertext operations.

Tier 2: embedding densities. The client embeds its documents with a public, frozen encoder and encrypts the mean embedding and second-moment matrix. The developer holds a Gaussian or log-linear density per source. Per-candidate scores then become plaintext–ciphertext quadratic forms of depth 2. This tier captures semantic similarity that n-grams miss, still without bootstrapping.

Tier 3: encrypted neural proxies. This is MixMin run directly. Small proxies use polynomial activations and attention in the style of PolyNeXt, which replaces activations with Hadamard products and polynomial attention kernels. The proxies are evaluated on encrypted target samples. We run this tier only on a subsample, as a reference for what full expressivity buys and what it costs.

For every tier we also record two kinds of information flow. The first is what the client can infer about the developer's sources from the decrypted scores: scores are linear in $C$, which the client knows, so the developer adds calibrated noise and limits $M$. The second is what the developer learns from the released mixture: a $K$-dimensional output protected with differential privacy. A separate analysis checks whether CKKS approximation error, a concern raised but untested in prior work on convex mixture objectives, ever reorders the top candidates.

Baselines

The do-nothing reference trains on the natural token proportions of the sources, and also on a uniform mixture. DoReMi reports that its optimized mixture reached the baseline accuracy 2.6x faster and improved average few-shot accuracy by 6.5 points over the default Pile mixture. That gap is the size of the prize. Every method gets the same final training budget, model size, and tokenizer.

The strongest existing method with full target access is plaintext MixMin. It serves as the oracle upper reference because it sees the target text in the clear. MixMin reports a 1–5% relative improvement in negative log-likelihood. We run it with its own per-source proxies, sized at its recommended 1% of final compute, on exactly the target samples the client encrypts. RegMix, whose reported Spearman correlation is 0.97–0.98, runs as a second plaintext oracle with the same proxy budget.

The success criterion is defined relative to these oracles. Tier 1 or Tier 2 beats the problem if it closes most of the gap between the natural mixture and plaintext MixMin on target negative log-likelihood. Concretely, we pre-register "at least half the gap" as the bar, a threshold we chose ourselves rather than a number from the file. It must do so while keeping encrypted cost within minutes of CPU time and one round.

The strongest competitor for the encrypted-target setting is DUET. Its reported results are GSM8K accuracy of 84.2 against 70.4 for training on Wikitext alone, after about 10 iterations. We give DUET the same sources and the same LoRA budget per iteration, and we match its client-side feedback channel by giving it the same differentially private noise level we use. We compare at matched wall-clock time and matched number of client interactions. We have no reported DUET result on our pre-training-style targets, so DUET is rerun rather than quoted.

The key ablation computes the identical Tier 1 and Tier 2 scores in plaintext. This separates two possible sources of failure: the cheap proxy itself, or the encryption and noise.

Why not just…?

Why not just send the proxy models to the client and let it run MixMin locally? This is the strongest alternative. When the developer is willing to ship per-source proxies, it removes encryption entirely, and we say so plainly. It fails when the source corpora are proprietary, because small models trained on a single source can leak that source's content. It also fails when the client cannot run model inference itself. Our Tier 1 and 2 release only noised inner products of source statistics. The plaintext ablation shows how much quality this costs relative to local MixMin, so a practitioner can price the trade.

Why not just have the client release differentially private n-gram counts and optimize in plaintext? This is a genuine threat when target sets are large and the hashed vocabulary is small. Noise added to a $V$-dimensional count vector grows with $V$. In our design, privacy is spent only on a $K$-dimensional decision, or on $M$ scores, so the advantage should widen as $V/K$ grows and as target sets shrink. We test this directly by sweeping target size from about 10^3 to 10^6 documents and $V$ across its range, at matched privacy budget. If released noisy counts match us across that sweep, the cryptographic layer is unnecessary for Tier 1, and we would report that.

Why not just use DUET-style black-box search? DUET needs a fine-tuning run per query and, by its own account, does not scale to pre-training mixtures. It also assumes the candidate training pools are plaintext and influence-scorable. Our comparison at matched wall-clock and interaction count will show whether one encrypted round can substitute for about 10 training-in-the-loop rounds.

Why not just optimize the mixture on public data and skip the client? The mixture that best fits a generic validation set need not fit a legal or clinical target. The do-nothing and public-target baselines measure this gap directly. If it is negligible for our targets, the setting itself is moot.

Proposed Experiments and Decisive Checks

Sources. We use the 22 domains of The Pile, and SlimPajama's sources as in MixMin, so our numbers sit beside published ones.

Simulated clients. Each "client" target is a held-out blend the developer never sees: single Pile domains such as PubMed or GitHub, random reweightings of several domains, and the Chinese Judge Case legal corpus from the encrypted-pretraining paper, which serves as a genuinely out-of-source target.

Training and metrics. Chosen mixtures train 150M and 1B models. Following Magnusson et al., we report target negative log-likelihood as the primary metric. Secondary metrics are rank agreement between encrypted-chosen and oracle-chosen mixtures, and encrypted cost: CPU time, ciphertext and key bytes, and rounds.

Implementation. Tier 1 and 2 run in CKKS with SIMD packing. Tier 3 runs on a 1,000-sample subset to keep cost bounded.

Decisive checks. Three results would settle the main claim. First, the plaintext Tier 1 ablation must capture a substantial share of MixMin's gain. If n-gram proxies recover almost nothing, the core claim is wrong, and Tier 2 becomes the candidate. Second, encrypted Tier 1 must pick the same top mixture as plaintext Tier 1 at standard CKKS precision, confirming that approximation noise does not reorder candidates. Third, across the target-size sweep there must be a regime where encrypted decision release beats released noisy counts at equal privacy. Together these three results either establish a cheap path to private mixture selection, or show precisely where it breaks.