When engineers search for new proteins or drug molecules, the objects of search are discrete strings that cannot be differentiated, so a common trick is to train a generative autoencoder — a network that compresses each string into a short continuous vector and decodes vectors back into strings — and then search continuously in that vector space. The catch is that the autoencoder's "latent" vector space is organized by string syntax, not by function. A small step in latent space may or may not change how useful the decoded molecule is, and the optimizer quickly wanders into regions where the decoder produces junk. CoBO measured this directly: in a reconstruction-trained autoencoder, the correlation between latent-space distance and objective difference is only about 0.2, which they raised to about 0.7 with an explicit Lipschitz penalty — but that penalty must be learned from expensive oracle evaluations of the objective itself.
At the same time, large pretrained foundation models, such as protein language models, already encode a great deal of functional knowledge in their internal representations. This proposal transfers that knowledge into the small latent space of a generative student, and the transfer mechanism is the crux: instead of forcing the student's latent codes to numerically match teacher coordinates — the rigid alignment that is known to over-constrain low-capacity students — we maximize a variational lower bound on the mutual information between teacher embeddings and student latent codes, following the estimator of VID. Because that bound carries a learned per-dimension variance, it automatically mutes whatever teacher dimensions the student cannot or should not reproduce. The core empirical claim is that this information-filtered transfer lets the latent space inherit the teacher's functional geometry before a single oracle query is spent, which should make subsequent latent search markedly more sample-efficient than either heuristic regularization or rigid alignment.
Core Claim
Three hypotheses, in increasing strength. First, rigid $L_2$ alignment of student latents to teacher embeddings should degrade latent navigability: it over-constrains a low-capacity encoder and distorts the geometry the optimizer needs, mirroring the capacity-overwhelm failure documented for coordinate-matching distillation by Huang et al.. Second, a variational mutual-information bound should raise the latent-to-objective distance correlation (CoBO's diagnostic) to roughly the level CoBO reaches with oracle-trained penalties, but at zero oracle cost, because teacher representation similarity already predicts functional similarity. Third, this geometry should translate into more high-fitness solutions found per oracle call under both gradient ascent and trust-region Bayesian optimization in latent space. Together these tests resolve a documented contradiction in the literature: distillation rigidity versus the Lipschitz smoothness that latent search requires — a tension that no paper in the file examines.
Research Program
We build one system, which we call information-distilled latent optimization (IDLO), in the protein domain where a strong frozen teacher exists (a pretrained protein language model). The backbone is the architecture of ReLSO: a ten-layer transformer autoencoder with a compact 30-dimensional latent bottleneck, which ReLSO showed can be navigated by exact gradient ascent once the landscape is regularized. IDLO changes only the training objective. For each training sequence $x$, the encoder produces a latent code $z$, and the frozen teacher produces an embedding $t$. Alongside the usual reconstruction loss we maximize a lower bound on the mutual information $I(T;Z)$ using a variational predictor $q_\phi(t \mid z)$, a Gaussian with learned per-dimension variance:
$$I(T;Z) \;\ge\; \mathbb{E}_{p(t,z)}\big[\log q_\phi(t \mid z)\big] + H(T), \qquad q_\phi(t\mid z) = \prod_k \mathcal{N}\big(t_k;\,\mu_k(z),\,\sigma_k^2\big).$$
The entropy term $H(T)$ is fixed by the frozen teacher, so only the expected log-likelihood is optimized; the learned variances $\sigma_k^2$ act as an automatic relevance filter, inflating on teacher dimensions that carry noise or structure the 30-dimensional bottleneck cannot express. This is exactly the mechanism Ahn et al. developed for distilling ResNets into small classifiers, transplanted from passive feature transfer into a generative, navigable latent space — a setting neither VID nor CRD ever touches. The theoretical footing is that of He et al., who proved that $L_2$ and cosine matching are merely restricted, uncalibrated lower bounds on mutual information; we use the calibrated bound instead, and we analyze when its optimum implies a Lipschitz-style link between latent distance and teacher-similarity, connecting to the metric that CoBO enforces by penalty. One extension guards against a known failure: raw mutual information can transfer label-irrelevant content, and the foundation-model distillation work concedes its generator becomes hostage to teacher blind spots. Where fitness labels exist (the GFP corpus of 51,175 labeled variants), we therefore also train a conditional variant $I(T;Z \mid Y)$ conditioned on fitness bins, borrowing the robustness rationale of Markov Knowledge Distillation. After training, the latent space is plugged unchanged into two search procedures: ReLSO-style gradient ascent through a differentiable latent fitness predictor, and the trust-region latent Bayesian optimization of LOL-BO, which is needed because Maus et al. showed global Gaussian processes fail outright once latent dimension reaches about 50.
Proposed Experiments
The primary benchmark is the ReLSO protein-fitness protocol on GFP (51,175 variants) and GB1 (160,000 variants), with a strict 60-step oracle budget and the count $\Phi$ of high-fitness sequences found as the headline metric; landscape ruggedness $\lambda_f$ and the latent-objective correlation $\rho$ (Pearson correlation between pairwise latent distances and objective differences, CoBO's diagnostic) are the mechanistic metrics, measured before any oracle call. On GFP, ReLSO reports $\Phi = 23$ versus at most 13 for hill climbing, MCMC, directed evolution, DbAS and CbAS, and a ruggedness drop from 8.42 in sequence space to 3.68 in its latent space; beating $\Phi = 23$ under the identical budget, or matching it with fewer oracle calls, is the success criterion. A second track uses the antimicrobial-peptide and malate-dehydrogenase generation tasks of the ESM2-distillation paper, evaluated with Macrel and APEX predictors; that paper reports roughly 97% of generated MDH sequences predicted functional where several baselines scored 0%, and no reported result found on Lacuna for its numeric AMP scores, so we re-run its baselines ourselves rather than compare against unstated scalars. A third track stresses scaling: the synthetic Ehrlich motif tasks of poli at sequence length 64, where discrete solvers ran out of memory in as few as 40 iterations and latent-space CMA-ES and SAASBO were the top performers in 128 dimensions. Finally, we track reconstruction validity throughout, since a distorted latent geometry is worthless if the decoder degrades — GVAE is the cautionary reference, with only 31% valid SMILES versus 17% for a character VAE.
Baselines
The "do nothing" reference is the same ReLSO-architecture autoencoder trained on reconstruction only, searched by unregularized latent gradient ascent — a configuration ReLSO shows produces gradients that fly off toward untrustworthy high-norm regions — plus random mutation search, which on GFP finds at most 13 high-fitness sequences in 60 steps per the ReLSO report. The strongest existing method on our exact benchmark is ReLSO itself, with its pseudo-concavity regularization (synthetic negative samples and interpolation penalties) and gradient ascent; it is the competitor to beat because it holds the best published number on this protocol, $\Phi = 23$ on GFP at a 60-step budget with ruggedness reduced from 8.42 to 3.68. We run it with its published architecture, which is also ours, the same teacher untouched, the same oracle budgets and the same tuning-search protocol (a fixed grid of equal size for every method), so any difference is attributable to the distillation objective alone. Beating it means $\Phi > 23$ on GFP, or equal $\Phi$ with strictly fewer oracle calls. The third competitor is a CoBO-style pairwise Lipschitz penalty grafted onto our architecture, trained on oracle-labeled pairs exactly as in CoBO; it is the strongest cheap alternative because it achieved the 0.2-to-0.7 correlation jump and top performance on seven GuacaMol tasks, and the decisive comparison is at matched small oracle budgets, where our claim is that the teacher supplies the metric for free while CoBO must spend queries learning it. The key ablation replaces the variational bound with plain $L_2$ alignment between latent codes and projected teacher embeddings, everything else identical; this isolates the adaptive-variance information filter as the active ingredient, and it doubles as a faithful replication of the distance-penalty spirit of the ESM2-policy-gradient work in a differentiable setting.
Why not just…?
Why not just minimize the $L_2$ distance between student latents and teacher embeddings? This is the simplest reading of "distill the teacher into the latent space," and it is essentially what the closest existing work does: the ProGen2/ESM2 paper steers a generator with a Euclidean latent-distance reward under policy gradients, which re-introduces exactly the geometric rigidity this proposal avoids, and its authors note the generator becomes hostage to teacher biases. The distillation literature is unambiguous that rigid matching fails under capacity mismatch: VID reached 85.18% distilling a wide CNN into an MLP where coordinate-matching methods plateau near 78%, and He et al. proved $L_2$ is a restricted lower bound of mutual information. But honesty requires saying this has never been tested inside a navigable latent space, where the failure mode would be geometric distortion rather than misclassified images; the $L_2$ ablation above measures $\rho$ and $\Phi$ directly, and if it matches the variational bound, the proposal's central mechanism is wrong and we will say so.
Why not just search directly in the teacher's embedding space? The teacher already encodes function, so why build a student at all? Because the teacher has no decoder: nothing maps an ESM2 embedding back to a synthesizable sequence, so search in teacher space cannot produce candidates, and even if it could, LOL-BO shows Gaussian-process search fails outright in embeddings of dimension 50 and up. The student autoencoder exists precisely to compress the teacher's knowledge into a space that is both low-dimensional and decodable; the question is only how to compress without destroying geometry, which is the proposal.
Why not just use CoBO's Lipschitz regularization, or skip latents entirely and run REINVENT or a genetic algorithm? These are the two real threats. CoBO's penalty is cheap and proven (correlation 0.2 to 0.7), so our defense is narrowly the cold-start claim: we predict high $\rho$ with zero oracle evaluations, and we will report the oracle-call count at which CoBO-style regularization catches up — if it catches up within a handful of queries, the teacher transfer adds little and the proposal should be judged accordingly. On model-free discrete search, the benchmarking studies cut both ways and we will say so: PMO found REINVENT ranked first overall and Graph GA second across 23 molecular tasks under a 10,000-query budget, so at large budgets simple search wins and we do not claim otherwise. Our target regime is the opposite: short budgets and long sequences, where ReLSO found model-free methods trapped at $\Phi \le 13$ versus 23 for latent ascent at 60 steps, and poli found discrete solvers exhausting memory at length 64. We will report performance across a range of budgets, including the regime where our method loses.
Decisive Checks
The proposal stands or falls on three numbers. If the variational bound lifts pre-search correlation $\rho$ from the ~0.2 of a reconstruction-only autoencoder to near CoBO's ~0.7 without oracle labels, the geometry claim holds. If $L_2$ alignment simultaneously lowers $\rho$ relative to reconstruction-only training, the rigidity contradiction is empirically confirmed, not just theorized. And if $\Phi$ on GFP exceeds 23 under the ReLSO protocol — or the MDH functional rate surpasses the ~97% reported by the ESM2-distillation paper with fewer predictor calls — the sample-efficiency claim holds. If any of the three fails, the fallback finding, cleanly documented, is still a negative result the two literatures currently disagree about: whether distillation helps or hurts a searchable latent space.
References
- Advancing Bayesian Optimization via Learning Correlated Latent Space
- Variational Information Distillation for Knowledge Transfer.
- Harmonizing Knowledge Transfer in Neural Network with Unified Distillation
- Transformer-based protein generation with regularized latent space optimization
- Contrastive Representation Distillation
- Distiller: A Systematic Study of Model Distillation Methods in Natural Language Processing
- Improving functional protein generation via foundation model-derived latent space likelihood optimization
- Markov Knowledge Distillation: Make Nasty Teachers Trained by Self-undermining Knowledge Distillation Fully Distillable
- Local Latent Space Bayesian Optimization over Structured Inputs
- A survey and benchmark of high-dimensional Bayesian optimization of discrete sequences
- Grammar Variational Autoencoder
- Sample Efficiency Matters: A Benchmark for Practical Molecular Optimization