Designing a protein sequence means searching a space of roughly $20^{100}$ candidates for a length-100 protein, using neural predictors as guides. The best search methods today, such as gradient-guided discrete Markov chain Monte Carlo (MCMC — a sampling scheme that wanders through a space, accepting or rejecting each step), query a large evolutionary model and several property predictors at every step, on a single device, with no notion of memory limits Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC. Meanwhile, the systems community has learned that when many parallel workers execute data-dependent work, some finish early and some stall, and that the fix — dynamically reassigning work to idle workers — is exactly what keeps hardware productive Piccolo: Building Fast, Distributed Programs with Partitioned Tables. And the hardware community has learned that GPU memory, not GPU arithmetic, is the binding constraint for large models, and that most of what sits in GPU memory is idle at any moment Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage. These three lessons have never been applied together to sequence design.
The proposal here does that, with a variational autoencoder (VAE — an encoder/decoder pair that compresses discrete sequences into a small continuous vector, its "latent space," and back) as the piece that makes it all fit. My core claim is that if you run the search in a VAE's latent space rather than over raw sequences, a search chain becomes a tiny, portable object — a few hundred floats plus a random-number-generator state — so a worker that finishes early can steal a chain from a busy worker at essentially zero migration cost. That same portability is what breaks existing memory-offloading systems, which assume a fixed, pre-profiled execution pattern; here, the scheduler itself knows which chains run next, so its assignment decisions double as prefetch hints telling the memory manager which model weights to stream into VRAM ahead of time. Search algorithm, load balancer, and memory manager become one feedback loop instead of three separate components.
Core Claim
Latent-space search over combinatorial sequence space can be made simultaneously statistically exact and cheap to rebalance at runtime, because migrating a chain costs only a vector copy, not a transfer of execution context. This lets a memory-constrained accelerator (a single GPU, or a few heterogeneous ones) run a large population of search chains through a large ensemble of guide models — an evolutionary prior at the scale of ESM-2 plus several property predictors — that would not fit in VRAM together. The concrete target is: match the design quality of the strongest single-device search methods on standard in-silico benchmarks while running on a fraction of the hardware, and beat static scheduling and stock offloading on chains-per-second at a fixed VRAM ceiling.
The search target follows the two strongest ideas in the design literature, combined in latent space. Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC multiplies an unsupervised evolutionary distribution $F$ by a supervised activity predictor $G$ into a single target; Proximal Exploration for Model-guided Protein Sequence Design adds a penalty on distance from the wild type to keep the search out of regions where the predictors are untrustworthy. I keep both terms but move the search variable from the discrete sequence $x$ to the latent vector $z$:
$$\pi(z) \;\propto\; \exp\!\Big( \log F(x_z) + \beta \log G(x_z) - \lambda\, d(x_z, x_{\mathrm{wt}}) \Big), \qquad x_z = \mathrm{decode}(z)$$
where $x_z$ is the sequence decoded from latent vector $z$, $F$ scores evolutionary plausibility, $G$ scores the desired property, $d$ measures mutational distance from the wild type $x_{\mathrm{wt}}$, and $\beta, \lambda$ trade off the three terms. Each chain runs a fixed transition kernel (Langevin or Metropolis-adjusted steps in $z$-space); every kernel step requires one decode plus one pass through $F$ and $G$.
The statistical claim is sharper than it may first appear. In Distributed Stochastic Gradient MCMC, adapting a worker's trajectory length to its speed changes the sampler and requires explicit bias-correction weights to preserve the target distribution. Here, stealing a chain does not touch the kernel at all: a chain's trajectory is fully determined by its latent position, its sampler state, and its random-number stream, so moving all three to another worker leaves the trajectory bit-identical. Only wall-clock progress is redistributed, so the stationary distribution $\pi(z)$ is preserved exactly by construction. If we additionally allow the population to adapt — cloning chains near fitness peaks and killing stuck ones, as population-based exploration wants — exactness is given up, and the honest position is to run both modes and measure the quality-versus-guarantee tradeoff empirically.
Research Program
The work has three components: a runtime, a memory manager, and a measurement of whether the load imbalance this whole design targets actually exists.
The runtime, which I will call LatentSteal, keeps a shared population of chains in a partitioned key-value table in the style of Piccolo: Building Fast, Distributed Programs with Partitioned Tables, where work stealing cut runtime nearly in half when one worker ran at half speed. A worker owns a set of chains, executes their kernels in micro-batches (decoding many $z$'s at once so GPU activations stay dense), and publishes each chain's state after every step. Idle workers pull chains from the longest queues. Two sources of skew make this non-trivial and are themselves an empirical question: chains decoded from an autoregressive VAE can produce sequences of different lengths, so per-step decode cost varies, and the multi-objective weighting can route different chains to different predictors in the ensemble with very different costs. Parallel and Scalable Short-Read Alignment on Multi-Core Clusters Using UPC++, facing exactly this data-dependent cost variance in genomic alignment, gained 42–50% over static assignment with dynamic on-demand task checkout — that is the magnitude of gain I expect if the skew is real, and if it is not, the first experiment will show it and the runtime story weakens to a scheduling nicety.
The memory manager is where the proposal departs most from existing systems. Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage shows that on average only 1.7% of allocated GPU memory is active at a given moment, and that streaming parked tensors directly between VRAM and NVMe drives (bypassing host DRAM) beat ZeRO-Offload and ZeRO-Infinity by an average of 1.47× — but the system requires a static, pre-profiled execution graph, and its authors' own analysis flags dynamic, data-dependent execution as an open failure mode. Latent search is exactly that failure mode: which guide model runs next depends on the scheduler's chain assignment, which depends on observed runtimes. The insight is to close the loop: the scheduler already decides, several milliseconds ahead, which chains execute next and therefore which weights and activations will be needed. Those assignment decisions are emitted as prefetch hints to the offloader. Dynamic execution, the thing that breaks static lifetime analysis, becomes the signal that makes prediction possible. This is the key ingredient the ablation must isolate (same runtime, prefetch hints disabled, pure reactive paging).
On the modeling side, the encoder/decoder pair can be the alignment-based MSA VAE of Generating functional protein variants with variational autoencoders, which captured long-range structural constraints and produced functional luciferase variants, or the unaligned autoregressive variant from the same paper, which handles variable-length sequences — the one route to indel exploration in this design, since ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design documents that alignment-locked models fail entirely on the 66 indel assays in its suite. The evolutionary term $F$ is a protein language model at ESM-2 scale, following Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC; it is the component whose memory footprint makes the memory wall real. The multi-objective machinery for the $G$ term follows evedesign: accessible biosequence design with a unified framework, which already packages conditional latent design under biophysical constraints. One honest limitation carried over from the VAE literature: the autoregressive decoder is biased toward local sequence patterns, so indel-capable runs may trade away some global-structure fidelity, and this tradeoff should be measured, not assumed away.
Proposed Experiments
The first experiment measures the problem before building the cure. I will run several thousand chain steps across the benchmark landscapes below with static scheduling, log the wall-clock cost of every kernel step, and report the skew (coefficient of variation of per-chain step cost, and idle fraction per worker). If per-step costs turn out near-uniform, the load-balancing half of the claim collapses to modest gains, and I would report that and recenter the proposal on the memory manager — this is a genuine risk, mainly because batched decoding amortizes length differences.
The second experiment measures design quality on the in-silico suites in the file. The primary suite is the TAPE-based set of eight optimization tasks from Proximal Exploration for Model-guided Protein Sequence Design (green fluorescent protein intensity, viral capsid viability, antibiotic resistance, and five more), scored by normalized fitness and mutation count. The secondary suite is the landscapes of Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC (PABP binding, UBE4B ligase activity, GFP fluorescence); no reported result found on Lacuna for that paper's per-task numerical scores, so it serves as a comparative setting against my own reimplementation of its method rather than against quoted numbers. Indel exploration is evaluated on the 66 indel assays of ProteinGym, where the reference point for prediction quality is ProteinNPT's Spearman correlation of 0.613; design quality on indel assays will be scored by the assays' own fitness labels, and I will state plainly if no comparable design baseline exists there.
The third experiment measures the systems claim: chains-completed-per-second and time-to-target-fitness (the wall-clock time until any chain finds a sequence exceeding a fixed fitness threshold) at a fixed VRAM ceiling, on one 24 GB-class GPU and on a small heterogeneous pool. Alongside chain throughput I will report GPU idle fraction, following the metric of EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference, which reduced GPU underutilization by mostly over 40% on skewed inference workloads — a comparable reduction under sequence-search skew is the systems success criterion. A final small-scale run on the luciferase family of Generating functional protein variants with variational autoencoders, whose conditional VAE produced 23 of 23 variants retaining measurable luminescence, would check that latent stealing does not degrade conditional generation; wet-lab validation is out of scope for this proposal.
Baselines
The do-nothing reference point is single-chain latent gradient ascent on one GPU with all models resident in VRAM, shrinking the model ensemble until it fits — this is what a practitioner does today and it bounds both quality (it can only run what fits) and throughput. The strongest existing design method on the primary benchmark is PEX with its Mutation Factorization Network from Proximal Exploration for Model-guided Protein Sequence Design, which reports an overall normalized score of 0.98 on the TAPE suite versus 0.89 for PEX with a plain CNN and 0.43 for AdaLead, and which reaches higher avGFP fitness in fewer than 10 mutations where baselines needed 20 to 50; it will be run with the same oracle models, the same fitness evaluation budget, and the same tuning effort as LatentSteal, on the same hardware where it fits and with its published setup otherwise. Beating PEX means matching its 0.98 overall score within noise while using strictly less VRAM, or exceeding it under the same budget. The closest algorithmic relative is the gradient-based discrete MCMC of Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC, which shares the Product-of-Experts target but searches sequence space on a single device; I reimplement it faithfully and compare at matched total model evaluations. On the systems side, the competitors are static batched execution with equal chain counts per worker, and static batching plus ZeRO-Offload for memory relief — the configuration that Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage beat by an average of 1.47× with direct-to-storage streaming, which sets the scale of improvement a well-built offloader should deliver. Finally, two ablations isolate the proposal's ingredients: LatentSteal with prefetch hints disabled (scheduler plus reactive paging only) isolates the co-design loop, and LatentSteal with chain stealing disabled but prefetch retained (static assignment, predictive offloading) isolates the balancing half.
Why not just…?
Why not just sample or prompt a pretrained protein language model directly, skipping the VAE entirely? This is the most serious objection, and the evidence against it is partly on its side: language models are excellent zero-shot scorers, and the top zero-shot substitution method in ProteinGym is an autoregressive model. But the design literature documents two recurring failures: unconstrained optimization of supervised predictors drifts into "white noise" sequences without an evolutionary guardrail, as reported in Plug & Play Directed Evolution of Proteins with Gradient-based Discrete MCMC, and discrete prompting gives no cheap, continuous handle for multi-objective steering, whereas the conditional VAE of Generating functional protein variants with variational autoencoders steered on solubility while keeping 23 of 23 variants active. Decisively for this proposal, though, the VAE is not mainly there for quality — it is there to make the chain portable, which a discrete sequence walk is not: moving a discrete-MCMC chain means migrating evaluation state tied to a specific position in a combinatorial space, while moving a latent chain is copying one vector. Experiment two includes a discrete-walk baseline (the reimplemented PPDE) precisely to test whether portability buys anything real.
Why not just shard the models statically across a cluster with standard parallelism (ZeRO-3, tensor/pipeline splits)? Static partitioning assumes predictable per-worker load, and the load-balancing literature shows this assumption fails badly under data skew: static expert placement in mixture-of-experts inference idled GPUs enough that EasyBalance recovered over 40% of utilization with dynamic cross-layer scheduling, and static task distribution in genomic alignment lost 42–50% of runtime to dynamic checkout in Parallel and Scalable Short-Read Alignment on Multi-Core Clusters Using UPC++. There is also an economic argument: TERAIO reports a 5.41× hardware cost reduction for a 70B model by streaming to NVMe instead of scaling out. Still, if a lab already owns a large homogeneous cluster, static sharding is a real threat to the practical value of this work; the fair response is empirical, running the static-sharding baseline at matched aggregate FLOPs and reporting the crossover point where dynamic balancing stops paying.
Why not just use classical directed evolution or genetic algorithms, which need no GPU at all? On shallow, additive landscapes this would genuinely work, and I will say so. The evidence against it on rugged landscapes is consistent: epistatic cliff-edges, where individually deleterious mutations are beneficial jointly, defeat greedy hill-climbing per Alon Wellner et al. and the biophysical-constraint analysis of Chen et al., and empirically AdaLead scored 0.43 overall on the TAPE suite against 0.98 for model-guided proximal search in Proximal Exploration, needing 20 to 50 mutations where model-guided search needed fewer than 10. AdaLead is already in the baseline set; if it matches LatentSteal on any landscape, that landscape is the wrong test case and the claim will be scoped to the rugged ones.
Why not just toggle ZeRO-Offload or ZeRO-Infinity and be done with the memory problem? Stock offloading routes tensors through host CPU DRAM, and TERAIO measured that this stalls the GPU badly enough that direct-to-SSD streaming won by an average of 1.47×; coarser-grained offloading also ignores that only 1.7% of allocated memory is active on average. More fundamentally, offloading frameworks assume regular, repeating execution, and search loops with acceptance/rejection branches and data-dependent predictor routing are precisely the case they handle worst. ZeRO-Offload is in the baseline set at matched settings, so the comparison will be direct rather than rhetorical.
Decisive Checks
Three results would decide the proposal's fate. First, the skew measurement: if per-chain step costs vary by less than roughly 20%, the load-balancing mechanism is not earning its complexity, and I would expect that honestly and retreat to the memory-manager contribution alone. Second, the quality check: LatentSteal must land within noise of PEX's 0.98 overall normalized score on the TAPE suite at matched evaluation budget — if latent-space search systematically underperforms discrete search, no systems gain rescues the method. Third, the systems check: at a fixed 24 GB VRAM ceiling, LatentSteal should complete strictly more chains per second than static scheduling with ZeRO-Offload, and disabling prefetch hints should reproduce a measurable fraction of the gap; if reactive paging alone matches the full system, the co-design loop — the one genuinely new mechanism here — is not doing the work, and the proposal's central claim fails with it.
References
- Plug & play directed evolution of proteins with gradient-based discrete MCMC
- Piccolo: Building Fast, Distributed Programs with Partitioned Tables
- Cost-Efficient LLM Training with Lifetime-Aware Tensor Offloading via GPUDirect Storage
- Proximal Exploration for Model-guided Protein Sequence Design
- Distributed Stochastic Gradient MCMC
- Parallel and Scalable Short-Read Alignment on Multi-Core Clusters Using UPC++
- Generating functional protein variants with variational autoencoders
- ProteinGym: Large-Scale Benchmarks for Protein Fitness Prediction and Design
- evedesign: accessible biosequence design with a unified framework
- EasyBalance: Cross-Layer Load Balancing in Distributed MoE Inference
- Rapid generation of potent antibodies by autonomous hypermutation in yeast
- Generalists vs. Specialists: Evaluating LLMs on Highly-Constrained Biophysical Sequence Optimization Tasks