Exact Full-Repertoire Gradients by Treating Receptors as Parallel Branches

Models that diagnose disease from a person's immune receptors have to find a needle in a haystack. One blood sample holds $10^5$ to $10^6$ distinct T-cell receptor sequences, and the handful that signal an infection or a tumor can make up less than 0.01% of them. The leading neural approach, DeepRC, pools all receptors with attention, a learned weighting that tells the model which sequences to look at. Storing the training activations for a million sequences does not fit on a GPU, so DeepRC computes gradients only for the 10% of sequences with the highest attention (Modern Hopfield Networks and Attention for Immune Repertoire Classification). Its successors instead subsample the repertoire randomly or drop repertoire-level training altogether. All of these shortcuts can throw away the rare signal sequences before the model has learned to notice them.

Work on scheduling memory in non-chain neural networks shows that when a computation branches and re-merges, the cost depends on where you cut it. This proposal uses that insight to show that the exact gradient over every receptor in a repertoire can be computed with memory bounded by a small chunk of receptors. For attention pooling this takes two passes. For lineage graph networks it takes a halo-recomputation schedule. We then use this tool for a question nobody has tested: does gradient truncation or subsampling actually cost accuracy and witness discovery, and in which signal regimes?

Core Claim

The memory-scheduling tools for branched networks, such as Rockmate, HiRemate, and the lower-set dynamic program of Kusumoto et al., reason over operator graphs. In those graphs the million receptors are just the batch dimension of one tensor. These tools therefore cannot see the structure that matters in a repertoire model. At the level of instances, the computation consists of $N$ independent encoder branches joined by one reducer. Gradient re-materialization, which means recomputing activations during the backward pass instead of storing them, works here exactly when the reducer's backward pass needs only a few global quantities.

For softmax attention pooling, let $s_i$ be receptor $i$'s attention score, $a_i$ its normalized weight, $v_i$ its value embedding, $z=\sum_i a_i v_i$ the pooled repertoire vector, and $g=\partial L/\partial z$ the loss gradient arriving at the pool. Then

$$\frac{\partial L}{\partial v_i}=a_i\,g,\qquad \frac{\partial L}{\partial s_i}=a_i\,(v_i-z)^\top g .$$

Every receptor's upstream gradient therefore depends only on its own quantities and on the global pair $(z,g)$. The schedule follows directly. The first pass streams chunks forward and keeps only $(s_i, v_i)$. The pool and loss are then computed. The second pass re-runs each chunk's encoder and back-propagates the gradient above. Peak activation memory is set by chunk size, not by $N$. The compute cost is about one extra forward pass, and the gradients equal those of untruncated training up to floating-point reordering.

For lineage graph neural networks like GRIP, message passing links receptors through edit-distance edges. A naive chunking then reproduces the "cut intersects a skip path" failure that Zhao et al. describe for skip connections. A chunk's $L$-layer outputs depend on its $L$-hop neighborhood in other chunks. Clonal families, however, form near-disconnected components. Partitioning along components, and recomputing a thin $L$-hop halo for families that are split, turns the instance graph back into a sequence of nearly separable blocks. This is the instance-level analogue of Rockmate's 1-separators.

Research Program

The first deliverable is a small PyTorch library for exact streaming training of repertoire models. It will implement the two-pass schedule for attention and Hopfield pooling, including multi-head pooling and the frequency-weighted aggregation used by LaDM³IL. It will also implement a component-aware partitioner for lineage graphs. The partitioner chooses chunks to minimize recomputed halo nodes under a memory budget, and it falls back to splitting very large public or expanded families with halo duplication.

The partitioning problem is where the DAG-scheduling view does real work. Among chunk orders, we restrict to those that respect the subclone-pooling hierarchy. These are lower sets in the hierarchy's sense: a subclone is pooled only after all its member receptors have been processed. Within that restriction, the cost is the number of recomputed halo nodes, which a greedy solver can minimize.

We will verify numerical gradient equality against full-memory training on repertoires small enough to fit, and report wall-clock overhead.

The second deliverable is the scientific study. We hold the architecture and hyperparameters fixed and train DeepRC-style models, plus a GRIP-style lineage model, under three gradient regimes. The first is exact full-repertoire gradients. The second is DeepRC's top-10% truncation. The third is random subsampling at matched compute. We measure repertoire classification and, separately, how well each regime recovers known witness sequences.

The hypothesis follows from the evidence in Profiling the baseline performance and limits of machine learning models for adaptive immune receptor repertoire classification that sequences per repertoire matter more than repertoire count at low witness rates. If that holds, exact gradients should help most when witnesses are rare and the signal is spread across many motifs. Truncation should hurt most early in training, when true witnesses have not yet earned high attention.

Baselines

The naive reference point is DeepRC exactly as published, with top-10% gradient truncation and 16-bit precision. It reports 0.831 ROC-AUC on the 685-repertoire cytomegalovirus (CMV) cohort, against 0.825 for a MinMax-kernel SVM and 0.699 for a Burden test (Modern Hopfield Networks and Attention for Immune Repertoire Classification). We re-run it on our splits with the same encoder, the same epochs, and the same hyperparameter search budget as the exact-gradient variant. The only difference is therefore which receptors receive gradient. This pair is the key ablation that isolates the proposal's ingredient.

A second ablation is matched-compute random subsampling. Each step samples enough receptors that the total compute equals the exact two-pass schedule. This tests whether exactness, rather than just seeing more sequences, is what helps.

The strongest published methods on CMV are NLL-IRC, which reports 83.24 AUC and 75.9 F1 using noisy instance labels and co-trained peer networks (A Noisy-Label-Learning Formulation for Immune Repertoire Classification and Disease-Associated Immune Receptor Sequence Identification). The other is LaDM³IL, which reports 84.88% AUC and 0.767 AUC for weakly supervised discovery of 164 known CMV receptors (A Label Disambiguation-Based Multimodal Massive Multiple Instance Learning Approach for Immune Repertoire Classification). LaDM³IL used 684 rather than 685 repertoires, so its numbers are not on identical splits. We will re-run both on our folds where code permits and otherwise quote them with that caveat.

Beating these methods is not required for the core claim, because they change the learning objective while we change the gradient. A clean win would be an exact-gradient model that exceeds the truncated version by more than the fold-to-fold standard deviation on CMV. A second clean win would be an exact-gradient LaDM³IL-style multimodal model trained at repertoire level that exceeds 84.88% AUC on matched splits.

On synthetic data, the reference is L1-penalized logistic regression on k-mers. It reaches 99% balanced accuracy at a 0.005% witness rate with a single motif but degrades with many motifs and gapped motifs (Profiling the baseline performance and limits of machine learning models for adaptive immune receptor repertoire classification). No numbers for the 64-motif setting were reported in the file, so we will generate them ourselves.

For the systems comparison, the baselines are off-the-shelf HiRemate and torch.utils.checkpoint applied to the traced model at the same memory budget. We compare the largest repertoire each method can train exactly.

Why not just…?

Why not just run HiRemate or Rockmate on the traced DeepRC graph? These tools partition operators, and in a traced repertoire model the million receptors live inside one tensor's batch dimension. The pooling node is therefore a single operator whose input must be fully materialized. Rockmate's 1-separator search and HiRemate's recursive partitioning have no branch to cut. We will measure this directly as the maximum repertoire size each tool trains exactly under a fixed 16 GB budget. If HiRemate matches our schedule, the systems contribution collapses and the study rests on the gradient-regime experiments alone.

Why not just wrap the encoder in chunked torch.utils.checkpoint? For the pure attention-pooling case this is a real threat. Our two-pass schedule is essentially chunked checkpointing made possible by the separability in the equation above, and we say so plainly. The non-trivial part is the lineage graph model, where chunk boundaries cut message-passing edges and plain checkpointing either produces wrong gradients or requires keeping the whole graph in memory. The equivalence tests against full-memory training will show which applies.

Why not just subsample, or keep DeepRC's top-10% truncation, since it already reaches 0.831? Possibly truncation is benign. Truncated DeepRC is within 0.01 AUC of NLL-IRC per the NLL-IRC paper, which reports DeepRC at 83.10. The matched-compute ablation and the witness-trajectory measurement below are designed to settle this. A null result, meaning truncation costs nothing even at low witness rates, would itself be useful, because it would tell the field the memory wall is not the bottleneck.

Why not just use L1 logistic regression or a kernel SVM? For single-motif signals they are enough, and the MinMax SVM sits at 0.825 on CMV. We include both. The claim we test is narrower: exact gradients matter specifically in the multi-motif, gapped, low-witness regime where the profiling study shows linear models break.

Proposed Experiments and Decisive Checks

The first check is correctness. On CMV repertoires downsampled to fit in memory, exact-schedule gradients must match full-memory gradients to within floating-point tolerance. We will also report peak memory and overhead as functions of chunk size and repertoire size up to $10^6$ receptors.

The second check is the decisive scientific one. We use the OLGA-based synthetic benchmark of the profiling study and DeepRC's synthetic suite, sweeping witness rate from 1% to 0.001% and motif count from 1 to 64, with gapped motifs included. For each regime we train identical models under the exact, truncated, and subsampled conditions across several seeds.

We report two outcomes. The first is classification AUC. The second is witness recall over training, defined as the fraction of implanted witnesses that ever enter the top-10% attention set during the first epochs. The prediction is that exact gradients improve AUC and recall mainly at witness rates at or below 0.01% with 16 or more motifs, and make no difference for single motifs. If the gap does not appear anywhere in this grid, the motivating hypothesis is false.

The third check uses real data: CMV with five-fold splits shared across all methods. We report ROC-AUC, F1, and discovery AUC against the known CMV receptors for exact versus truncated DeepRC and for an exact repertoire-level version of the LaDM³IL multimodal model.

The fourth check tests the lineage model. CMV's TCRβ repertoires lack somatic hypermutation, and TCGA repertoires recovered via TRUST are shallow, so we will build edit-distance graphs on synthetic repertoires with implanted clonal families. We then compare a GRIP-style model trained exactly at full depth against the same model trained on the subsampled depth that fits without our schedule. On TCGA we expect no benefit, since repertoires are already small. A gain there would therefore suggest a bug rather than an effect.