Tolerance Dropout: Learning What to Ignore in T-Cell Receptor Repertoires

Every person's blood carries millions of distinct T-cell receptor (TCR) sequences — a "repertoire." Machine-learning models that read repertoires must solve a problem that is, structurally, the same one the immune system itself solves: stay silent toward the overwhelming background of self-reactive and bystander receptors, while remaining responsive to the handful of rare clonotypes that actually signal a past infection. Current repertoire models fail at exactly this. The strongest existing approach builds a dense embedding for each TCR from co-occurrence statistics and then averages all of them into one patient-level vector (Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires); the authors themselves report that as background receptors multiply, this averaging dilutes weak disease signals into noise. Meanwhile, the dropout literature has spent a decade documenting the mirror-image failure: uniform dropout drops rare, high-signal features at the same rate as redundant ones (Adaptive dropout for training deep neural networks), and learnable discrete dropout (Learnable Bernoulli Dropout for Bayesian Deep Learning) can now push uninformative units to zero retention automatically — but has never touched a repertoire dataset.

This proposal unites the three directions by making each do the job it is uniquely suited for. Adaptive dropout optimization supplies the core mechanism: per-clonotype Bernoulli gates, trained with unbiased discrete gradients, that learn to tolerate bystander receptors and retain informative ones. Uniform stochastic regularization supplies the diagnosis: why a single shared drop probability cannot work on heavy-tailed, noise-dominated repertoires. And immune tolerance induction supplies both the application domain and a design principle that no current learnable-dropout method has — the rheostat, the coupled negative feedback that keeps suppression and responsiveness in balance rather than letting gates collapse to "ignore everything" or saturate at "ignore nothing." The result is a model that learns tolerance the way the biological system must: as an active, dynamically balanced suppression decision, not a global mute.

Core Claim

Learnable Bernoulli gates over pre-trained TCR embeddings, trained with the Augment-REINFORCE-Merge (ARM) gradient estimator and stabilized by an explicit occupancy constraint modeled on the regulatory/effector rheostat, will produce patient-level repertoire representations that (i) match near-ceiling performance where current methods already succeed, (ii) substantially outperform mean pooling and uniform dropout exactly where current methods fail — rare-signal detection under heavy background noise — and (iii) yield gates whose learned retain rates align with ground-truth antigen specificity, making "tolerance" an inspectable, validated quantity rather than a metaphor.

The claim is sharply falsifiable. If learned gates do not beat uniform dropout on a background-dilution stress test, or if gates driven to zero correspond to genuinely informative clonotypes rather than bystanders, the biological story is decoration and the mechanism should be abandoned.

Method: Tolerance-Inducing Learnable Dropout

For a repertoire of $n$ clonotypes with pre-trained embeddings $x_1,\dots,x_n \in \mathbb{R}^d$ (reused from the co-occurrence pre-training of Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires, trained on the TDETECT and PUBLIC cohorts of up to roughly 32,000 repertoires), draw an independent Bernoulli gate $m_i \sim \mathrm{Bernoulli}(\pi_i)$ per clonotype and pool only the retained ones: $z = \big(\sum_i m_i x_i\big) / \max(1, \sum_i m_i)$, then classify $z$ with a small head $g$. The retain rate of clonotype $i$ is

$$\pi_i = \sigma\!\big(\alpha_i + \beta \log(1/f_i)\big),$$

where $\sigma$ is the logistic function, $\alpha_i$ is a learned logit, $f_i$ is the clonotype's frequency within the repertoire, and $\beta$ is a learned masking strength. This is the antigen-masking analogue: abundant bystander clones begin with depressed retention, just as high-affinity antibodies mask antigen and starve lower-affinity clones of survival signal during germinal centre termination (Germinal Centre Shutdown). Unlike GI-Dropout, which hides dominant features using global dataset statistics for vision tasks (From Random to Supervised: A Novel Dropout Mechanism Integrated with Global Information), the prior here is per-repertoire, frequency-driven, and refuted or confirmed by gradient learning on top of it.

The second ingredient is the rheostat, motivated by a concrete optimization failure. Unconstrained learnable rates are known to be unstable: early variational dropout with improper log-uniform priors underfit or collapsed, requiring hierarchical proper priors to stabilize (Kingma et al., 2015; Liu et al., 2019), and nothing stops all gates in a repertoire from collapsing toward zero (representation starvation) or one (no tolerance at all). Biology maintains balance not with static thresholds but with coupled feedback — regulatory T cells with high-affinity IL-2 receptors constitutively sequester the growth factor that effector cells need (Yingyue Zhang et al.); the effector/regulatory ratio, not either population alone, is the controlled variable (Research progress of traditional Chinese medicine...; Hypothalamic kisspeptin alleviates myasthenia gravis...). The direct translation is an occupancy constraint on the aggregate gate, leaving individual gates free to compete for it:

$$\mathcal{L} = \mathbb{E}_{m}\big[\ell(g(z))\big] \;+\; \lambda\Big(\frac{1}{n}\sum_{i=1}^{n}\pi_i - \rho\Big)^{\!2},$$

where $\ell$ is the classification loss (e.g., cytomegalovirus serostatus), $\rho$ is the rheostat set-point (the target fraction of retained clonotypes), and $\lambda$ is the coupling strength. A gate can be driven to zero only by taking retention budget from elsewhere — local competition for a global survival signal, as in cytokine sequestration. Gradients with respect to $\alpha_i$ flow through the discrete masks using the ARM estimator, which computes exact, unbiased gradients by pairing two correlated gate draws and needs no continuous relaxation (Learnable Bernoulli Dropout for Bayesian Deep Learning); this matters because relaxations like Concrete dropout introduce gradient bias and cannot produce hard zeros, i.e., cannot actually silence a clone. There is also a theoretical reading: dropout training is provably equivalent to minimax optimization over a distributional uncertainty set (Dropout Training is Distributionally Robust Optimal), so per-clonotype learnable rates amount to spending a fixed robustness budget non-uniformly — on bystanders rather than on rare signal clonotypes, which is the whole point.

One design warning comes from the immunology itself: in lupus, regulatory cells are present in normal numbers but lose suppressive function (Shihan Yu et al.). The gate analogue is rates hovering near 0.5 — abundant but undifferentiated. We will therefore evaluate not just mean retention but gate bimodality (the fraction of gates committed near 0 or 1) as the machine-learning counterpart of functional, not merely numerical, tolerance.

Research Program

The program has three parts. First, build the model: a PyTorch implementation on top of an existing variational-dropout codebase (MetaVD), with the ARM-gated pooling head, the frequency-based masking prior, and the occupancy constraint; pre-trained TCR embeddings are reused as-is so that all gains are attributable to gating. Second, analyze stability: prove (or empirically characterize) when the unconstrained objective admits degenerate solutions — global gate collapse or saturation — and show the occupancy constraint eliminates them, connecting this formally to the prior-prior stabilization of variational dropout (Liu et al., 2019) and to the flat-minima argument used by continual variational dropout (Continual variational dropout: a view of auxiliary local variables in continual learning). Third, as a secondary question, test whether tolerance gates driven by prediction uncertainty rather than task labels remove the explicit task-boundary requirement that limits continual variational dropout to known, discrete tasks — the immune system tolerizes continuously without external task identifiers (LPX-TI641, a Tim3/4 Agonist, Induces Long-Term Immune Tolerance in Multiple Sclerosis Models). This third part is deliberately last and optional; it is not a claim the proposal stands on.

Baselines

The naive reference point is mean pooling of pre-trained TCR embeddings, the method of Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires, which reports CMV detection AUROC of approximately 0.99 on the EMERSON cohort — near ceiling — while degrading under background noise on Parvovirus (no reported number found on Lacuna for the Parvovirus AUROC). Beating it on CMV is not the bar; matching it there while beating it under dilution is.

The strongest existing competitor on this exact question does not yet exist as a published result: no reported result found on Lacuna for uniform dropout applied to TCR repertoire embeddings. We therefore run it ourselves — grid-searched uniform Bernoulli dropout at the pooling layer over $p \in \{0.1, 0.2, 0.3, 0.5, 0.7\}$ — with the same pre-trained embeddings, the same classifier head, the same training data, and the same random-search tuning budget (fixed number of trials and wall-clock cap) as every method. Because this baseline is computationally free, tolerance dropout must justify its overhead; we will report training cost per epoch honestly against it.

Two further baselines isolate the proposal's ingredients. Concrete dropout — a continuous Gumbel-style relaxation that learns rates by plain backpropagation — is the simplest learnable alternative; on CIFAR-10 with VGG19 it reaches 92.72% accuracy versus 93.47% for discrete learnable dropout (Learnable Bernoulli Dropout for Bayesian Deep Learning), so it is a serious, near-parity competitor. And learnable Bernoulli dropout applied naively — per-clonotype gates with ARM but without the masking prior and without the occupancy constraint — is the critical ablation: any performance gap between it and tolerance dropout is attributable solely to the two biologically motivated ingredients, and if that gap is zero the rheostat story adds nothing. A deterministic attention-pooling head (channel-attention-style soft weights over clonotypes) completes the set, since it can learn per-clone importance without any stochastic machinery.

What counts as winning: on Parvovirus detection and on the dilution stress test described below, tolerance dropout must exceed mean pooling, uniform dropout, Concrete dropout, and attention pooling in AUROC at matched tuning budget; on CMV it must stay within noise of the reported ~0.99; and the ablation against naive learnable dropout must show a measurable, reproducible gap on at least one of these. Failing any leg of that triad counts as a negative result for the proposal.

Why not just…?

Why not just grid-search uniform dropout? It is free, robust, and would be the first thing a practitioner reaches for. But it is structurally wrong for repertoires: uniform rates drop the rare disease-specific clonotype with exactly the same probability as the thousandth bystander clone — the information-starvation failure documented by Adaptive dropout for training deep neural networks and Adaptive Dropout Rates for Learning with Corrupted Features. And where batch-normalized architectures are involved, uniform dropout can actively hurt, dropping a ResNet-18 Tiny ImageNet baseline from 48.8% to 47.4% accuracy (Interpreting and Boosting Dropout from a Game-Theoretic View). That said, the dilution stress test is where this settles empirically: if uniform dropout with a well-chosen $p$ decays as slowly as tolerance dropout under background spike-in, uniform wins on cost and we should say so.

Why not just use Concrete dropout and backprop? It is genuinely simpler and the closest credible rival. The evidence against it: continuous relaxations carry gradient bias and posterior distortion, they trail the discrete ARM approach at parity (92.72% vs. 93.47% on CIFAR-10, and 81.17% vs. 85.16% mean PAvPU uncertainty calibration on CamVid, all from Learnable Bernoulli Dropout for Bayesian Deep Learning), and — decisive for our setting — soft multipliers never produce hard zeros, so a bystander clone is never fully silenced, only attenuated, leaving residual dilution in the pooled vector. We run Concrete dropout as a baseline anyway; if it matches tolerance dropout under dilution, the ARM machinery is unjustified overhead and we will report that.

Why not just use deterministic attention or feature selection over clonotypes? This is the most serious alternative, and we treat it as such. Soft attention can learn per-clone importance scores without sampling, without variance, and without biological machinery. The literature's counterarguments are that deterministic weighting does not break co-adaptation among features and gives up the distributionally robust optimum that dropout training provably solves (Dropout Training is Distributionally Robust Optimal); Interpreting and Boosting Dropout from a Game-Theoretic View shows overfit samples carry disproportionately strong feature interactions that deterministic scaling leaves intact. But those arguments were made on vision benchmarks, not repertoires. The honest resolution is empirical: attention pooling is a first-class baseline in every experiment, and the dilution test plus the MIRA gate-validation study below directly measure whether stochastic gating buys anything soft weights do not.

Why not just train sequence models (CNNs or Transformers) directly on receptor sequences? Because TCRs are generated by somatic V(D)J recombination — a semi-random assembly process — rather than by descent, so sequence similarity does not track antigen specificity the way it does for evolved proteins; functionally related receptors targeting the same antigen can be nearly dissimilar in sequence (Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires). Sequence-only supervised models also face millions of mostly-unlabeled clonotypes per patient and small labeled clinical cohorts, a regime in which they overfit badly. This is why the proposal builds on co-occurrence embeddings and attacks the pooling layer, where the noise problem actually lives.

Decisive Experiments

The primary benchmark is the EMERSON and MULTIID cohorts with disease-exposure labels, exactly as used by Scalable Universal T-Cell Receptor Embeddings from Adaptive Immune Repertoires: CMV detection (their reported AUROC of approximately 0.99 is the number to match), Parvovirus detection (where their mean pooling degrades under background noise; no reported AUROC found on Lacuna), and HLA genotype classification across 145 alleles. The single most decisive experiment is the dilution stress test: spike $k \times$ unrelated background TCRs drawn from the healthy PUBLIC cohorts into each EMERSON repertoire at $k \in \{1, 2, 5, 10, 20\}$, and compare the AUROC decay curves of all methods under identical spike-ins. Tolerance dropout should show a markedly shallower decay slope; the claim is quantified as the area between decay curves, with uncertainty from bootstrap over subjects.

The second decisive experiment validates tolerance itself against ground truth using the MIRA assay, which confirms which TCRs bind which antigens: gates learned on CMV-positive repertoires should have systematically higher retain rates for MIRA-confirmed CMV-specific clonotypes than for matched bystanders, measured as the AUROC of gate values as a predictor of confirmed specificity. This turns the immunological framing into a testable statement about the model's internals rather than its accuracy alone.

Third, stability and ablation studies: across training runs, log the incidence of repertoire-level gate collapse (mean $\pi$ below 0.05) and saturation (above 0.95) with and without the occupancy term, plus the gate-bimodality statistic; report training cost per epoch against every baseline. Finally, a compact general-domain sanity check on CIFAR-10 with VGG19 (against the 93.47% accuracy reported by Learnable Bernoulli Dropout for Bayesian Deep Learning) confirms the occupancy constraint does not damage the ordinary learnable-dropout case where it is unnecessary.

The negative-result criteria are stated up front: if tolerance dropout does not beat uniform dropout and attention pooling under dilution, or if MIRA validation shows gates silencing genuinely specific clonotypes, then the learned "tolerance" is not tolerance, and the finding — that rheostatic feedback adds nothing to learnable dropout on noise-dominated biological data — will be reported as such.