Capacity-Adaptive Public/Private Caching for Immune Repertoire Attention

An immune repertoire is the set of ten to a hundred million receptor sequences that encode each person's adaptive immunity. Learning to diagnose infection or autoimmune disease from a repertoire is a needle-in-a-haystack problem: in a cohort of hundreds of patients, each with a bag of millions of sequences, the disease signal lives in a tiny subset of "witness" sequences, and the label is attached only to the whole bag. The best current models score every sequence with attention — Modern Hopfield Networks and Attention for Immune Repertoire Classification (DeepRC) and A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach for Immune Repertoire Classification (LaDM³IL) — and both hit a wall: they must hold millions of feature vectors per patient in GPU memory. Separately, network information theory has solved an abstractly identical problem for correlated content: split a library into a shared part and per-item private parts, and allocate scarce cache memory between them according to capacity. And immunology has already measured the relevant statistic: only about 10% of disease-driving T-cell receptor clonotypes are "public" (shared across unrelated patients), while 90% are private, patient-specific, and — remarkably — persist for decades (Disease-driving CD4+ T cell clonotypes persist for decades in celiac disease). No one has connected these three facts.

This proposal builds an attention memory for repertoire learning that decomposes each patient's sequences into a cohort-shared motif dictionary plus small per-patient private residuals, and — this is the key ingredient — allocates memory between the two using a capacity-dependent controller rather than a fixed heuristic. The scientific payoff is twofold. Practically, we target attention memory below 1% of the full footprint while holding accuracy at the level of the strongest baselines. Theoretically, the caching literature disagrees with itself about what scarce memory should store: Rate-Memory Trade-Off for Caching and Delivery of Correlated Sources proves that low-capacity caches must prioritize private descriptions, while Coded Caching With Heterogeneous User Profiles shows the opposite ordering holds in its multicast setting. Which regime structure real biological repertoires exhibit is an open, measurable question.

Core Claim

The claim has a method half and a science half. Method: a decoupled attention memory for multiple-instance learning on immune repertoires — a shared, learned motif pool playing the role of the Gray-Wyner common description, plus per-patient private residual caches — can preserve rare private witness sequences at a memory budget where uniform pruning destroys them, because it suppresses the redundant public background selectively instead of non-specifically. Science: the optimal memory split between the shared pool and private caches is a function of total memory with an empirically measurable crossover threshold, and that threshold is not predicted by the raw information shares. The 90/10 public/private split measured in repertoires (Disease-driving CD4+ T cell clonotypes persist for decades in celiac disease) does not dictate a 10% private allocation, just as the 90% shareable-content finding in CollectiveKV: Decoupling and Sharing Collaborative Information in Sequential Recommendation does not dictate a 90% shared pool under every budget.

There is a deliberate design analogy here with selective immunosuppression, the third direction. Top-$k$ attention pruning is the computational analog of broad-spectrum immunosuppression: it suppresses everything indiscriminately, discarding rare pathogenic witnesses together with background noise. The immunology literature documents the cost of that strategy and the payoff of selective ones — Application and prospect of targeting innate immune sensors in the treatment of autoimmune diseases catalogs the relapses and opportunistic infections of downstream cytokine blockade, while selective agents spare host function: the cGAS inhibitor XQ2b preserves RNA-virus sensing while fully protecting Trex1−/− mice (Development of cyclopeptide inhibitors of cGAS targeting protein-DNA interaction and phase separation), and the TNFRSF17-directed engager IBI379 kills pathogenic plasma cells while sparing normal B cells (Uncovering lupus nephritis-specific genes and the potential of TNFRSF17-targeted immunotherapy). Our private cache plays the role of the spared compartment: it is where rare, patient-specific, disease-driving clonotypes survive compression. The measurable version of "sparing protective immunity" is witness-receptor identification accuracy on known CMV-reactive receptors.

Research Program

The work has three components. First, a repertoire decomposition: we encode each patient's sequences with the multimodal features LaDM³IL showed to be necessary — V/J gene usage fused with amino-acid embeddings, since sequence similarity alone misses V(D)J context (A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach) — and cluster cohort sequences into a public motif dictionary plus quantized private residuals. On this decomposition we estimate the empirical rate structure: bits per sequence for the shared dictionary versus private residuals at several distortion levels, the quantity whose theoretical counterpart bounds delivery in Rate-Memory Trade-Off for Caching and Delivery of Correlated Sources (where the achievable gap is at most half the conditional entropy). This is the first empirical measurement of a Gray-Wyner-style rate structure on real repertoires, closing the gap that the theory has only been characterized for synthetic two- and three-file libraries.

Second, the architecture. We adapt the CollectiveKV design — shared high-dimensional pool, tiny per-user projection, router with peak and load-balance losses (CollectiveKV) — to this decomposition, replacing the recommendation click-stream encoder with the repertoire encoder. The per-patient memory becomes the private residual cache; the pool stores dictionary entries. During attention, each sequence retrieves its dictionary entry and adds its private residual if one is cached, otherwise reconstructing from the shared entry alone at some distortion.

Third, the allocation controller, which is what no prior system has. Let $K$ be the number of patients in a batch, $|\mathcal{D}|$ the number of dictionary entries of dimension $d$, and $M_k$ the per-patient private cache budget:

$$|\mathcal{D}|\,d \;+\; \sum_{k=1}^{K} M_k \;\le\; M, \qquad \rho \;=\; \frac{\sum_k M_k}{M}$$

where $M$ is the total memory budget and $\rho$ is the fraction devoted to private residuals. The controller chooses $\rho$ (and dictionary size) as a function of $M$. The Gray-Wyner result predicts $\rho$ should be high at small $M$ — cache private first, share the rest — while the heterogeneous-profiles result predicts a threshold of the form $M \ge N_c - GN_u/K$ below which shared content wins (Coded Caching With Heterogeneous User Profiles). We treat the direction and sharpness of the crossover as the object of study: we measure $\rho^*(M)$ empirically, then test whether the measured threshold is predicted by the estimated rate structure. Layering gives a second knob in the same spirit: the dictionary can store coarse sequence features as a base layer and fine structural detail as refinement layers, and the ordered-versus-proportional allocation contrast from Centralized Coded Caching for Heterogeneous Lossy Requests — fill the base layer completely before refinements, versus split proportionally — maps directly onto how dictionary resolution should grow with budget. That paper reports that proportional allocation tracks the cut-set bound with an approximately constant gap but gives no single aggregate scalar, so we will rely on our own frontier measurements rather than a published number.

Proposed Experiments

The primary benchmark is the CMV cohort of 684 individuals used by both DeepRC and LaDM³IL, where reported numbers anchor every comparison: DeepRC reaches ROC-AUC 0.831, an SVM with a MinMax kernel 0.825, and a burden test 0.699 (Modern Hopfield Networks and Attention for Immune Repertoire Classification); LaDM³IL reports 84.88% AUC on the same task (A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach). We sweep the memory budget $M$ and report the full AUC-versus-peak-GPU-memory frontier for every method, plus wall-clock latency (CollectiveKV's 32.99 ms to 0.69 ms reduction on an A100 shows what decoupling can deliver in recommendation; we measure the repertoire analog ourselves, since no reported result exists on Lacuna for repertoire-model latency). Three decisive comparisons structure the sweep: whether our method matches LaDM³IL's 84.88% at an order of magnitude less memory; whether it beats fixed-split decoupling at small $M$; and whether $\rho^*(M)$ shows a sharp crossover. The witness-identification task — recovering the 164 known CMV-reactive receptors, where LaDM³IL reports AUC 0.767 — is our selectivity assay: private caches should preserve rare witnesses that pruning discards. The low-witness stress test uses the synthetic 0.1%-witness setting, where naive methods collapse toward chance while DeepRC holds ROC-AUC 0.984 only by keeping all sequences in the forward pass (Modern Hopfield Networks and Attention for Immune Repertoire Classification); we test whether decoupled memory holds comparable accuracy at a fraction of that footprint. Finally, on the 21-patient longitudinal celiac dataset with up to 53% clonotype overlap across samples 16–28 years apart (Disease-driving CD4+ T cell clonotypes persist for decades in celiac disease), we check whether decades-persistent private clonotypes survive in the private cache — a biological validation that the private compartment captures meaningful signal rather than quantization noise.

Baselines

The do-nothing reference is full-memory attention: every sequence of every patient in the batch held as a feature vector, DeepRC-style but without any memory reduction. It is the accuracy anchor and the memory worst case, and it costs nothing to run alongside the others on identical splits.

The strongest memory-heuristic competitor is DeepRC itself, which reduces gradient-time memory by updating on only the top 10% of sequences ranked by attention weight, and reports ROC-AUC 0.831 on the CMV cohort (Modern Hopfield Networks and Attention for Immune Repertoire Classification). We run it as released, on our splits, with an equal tuning budget, and measure peak memory and latency alongside AUC. Beating it means matching its AUC at a tenth of its peak memory — its pruning retains 10% of the sequence volume by construction, while the decoupled design targets the sub-1% regime that CollectiveKV reached in recommendation (compression rate 0.008).

The strongest accuracy competitor is LaDM³IL, which reports the best CMV classification AUC in the file at 84.88%, 90.22% on cancer receptor identification, and 0.767 on witness identification (A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach). It is the strongest competitor on the scientific question because it already addresses supervision ambiguity — the same noise problem our private cache attacks — through label disambiguation and prototype decoupling rather than memory structure. We run it with the same cohort, the same train/test splits, and a matched hyperparameter-search budget, and re-implement its scorer inside our memory measurements so the comparison isolates memory architecture rather than encoder differences. Beating it means matching its AUC within noise at an order-of-magnitude smaller footprint and clearly winning at constrained budgets.

The critical ablation is a fixed-split port of CollectiveKV: the same shared pool and private projection machinery, but with the split frozen at the hand-tuned values that worked for recommendation (roughly 90% shared, user-specific dimension around 4% of the original). This isolates exactly one ingredient — capacity-adaptive allocation. If the adaptive controller does not beat this ablation somewhere on the memory sweep, the central claim of the proposal fails. A second ablation, an uncoded cache of the most frequent public motifs, serves as the naive-caching reference point.

Why not just…?

Why not just prune attention to the top 10%? Because pruning is non-specific by construction, and the evidence shows where that fails: naive subsampling collapses toward random guessing at 0.1% witness rates, and DeepRC survives that regime only by keeping every sequence in the forward pass, so its inference memory stays linear in repertoire size (Modern Hopfield Networks and Attention for Immune Repertoire Classification). Ranking-based pruning also presumes the attention scores that identify witnesses are already accurate — circular for exactly the rare private clones the model has seen least. That said, at generous memory budgets pruning is a genuine threat, and our first experiment will show it plainly on the frontier plot; our claim is scoped to the low-memory, large-cohort regime.

Why not just cache the most frequent public motifs with standard LRU/LFU? Because the caching theory in the file says frequency-first is provably suboptimal in exactly one scarce-memory regime and empirically best in another, and the disagreement is unresolved. Rate-Memory Trade-Off for Caching and Delivery of Correlated Sources proves private-first allocation at low capacity and achieves zero-traffic states with caches up to 40% smaller; Coded Caching With Heterogeneous User Profiles shows shared-first schemes within a factor of 8 of optimal in its setting, with unique content worth caching only above an explicit threshold. Neither result covers biological statistics, so we do not argue this one out — the $\rho^*(M)$ sweep is designed to settle it, and the LFU baseline is in the comparison set.

Why not just port CollectiveKV with its existing 90/10 split? This is the most serious alternative, and we treat it as the make-or-break ablation rather than an argument. Two facts suggest the ported defaults are wrong for repertoires: the 90/10 shareable split was measured on recommendation click streams, whereas celiac repertoires show the inverse — about 10% public clonotypes (Disease-driving CD4+ T cell clonotypes persist for decades in celiac disease) — and CollectiveKV's own authors flag that its split is a manually chosen heuristic, not derived from capacity limits. If the fixed split matches our adaptive controller across the entire memory sweep, our capacity-adaptation claim is wrong and we will report that; but the inverted public fraction and the theory's predicted regime shifts both give reason to expect the optimum to move.

Decisive Checks

The proposal stands or falls on three numbers. First, the frontier: matched AUC (against LaDM³IL's 84.88% CMV result) at peak memory below 1% of full storage, with the crossover in $\rho^*(M)$ located and compared against the threshold predicted from the measured rate structure. Second, selectivity: witness-receptor identification AUC at or above the 0.767 LaDM³IL reports, demonstrating that rare private clonotypes survive compression. Third, the ablation verdict: adaptive allocation must beat the fixed-split CollectiveKV port somewhere on the sweep, and must beat top-10% pruning at small budgets. If the fixed split wins everywhere, the honest conclusion is that capacity adaptation is unnecessary for repertoires and the interesting residue is the first empirical Gray-Wyner rate measurement itself — a smaller but still publishable result. If neither beats pruning anywhere, the premise that decoupled memory helps repertoire learning is falsified, and we would report the negative finding together with the measured rate structure, which would then stand as evidence against transferring caching-theory structure to biological sequence libraries.