Alignment Supervision for Free: Self-Supervised Edit-Aware Pre-Training of Biological Sequence Encoders

Biological sequences such as DNA, proteins and T-cell receptors differ from one another mainly through edits: substitutions, insertions and deletions. A model that understands these sequences should know which position in one sequence corresponds to which position in a related one. The classical way to compute that correspondence is Needleman-Wunsch alignment, a dynamic program that fills an $n \times m$ table. Its cost grows with the product of the two lengths, which is why recent work uses the table only as a training-time teacher. Needleman-Wunsch Attention (NWA) turns the alignment table into a target and pushes a Transformer's cross-attention toward it. That improved edit-distance embeddings by 13% on synthetic DNA. It also has two costs: it pays for the full table on every training pair, and it needs pairs of related sequences to begin with. Most domain corpora used to pre-train specialized models, such as the roughly 88,000 unlabeled receptors behind TCR-BERT, are collections of single sequences with no partner.

Our proposal starts from one observation. During pre-training we create the corrupted sequence ourselves, so we already know the true alignment between the original and its edited copy. It is a by-product of the corruption, recorded in linear time, and no dynamic program is needed. We propose to pre-train domain sequence encoders by applying sampled insertions, deletions and substitutions to each unpaired sequence. The encoder then has two jobs: reconstruct the masked content, and make its cross-attention between original and copy follow the recorded edit path. This turns alignment supervision from a paired, quadratic-cost teacher into a self-supervised objective that costs nothing extra and runs on any single-sequence corpus. That combination is the new piece.

Core Claim

The claim has two parts. First, the benefit NWA showed from alignment-shaped attention can be kept, and probably increased, when the teacher is the known corruption path instead of a computed optimal alignment. Once the teacher is free, the method extends to long sequences where NWA's exact table is too expensive, and to unpaired corpora where NWA cannot run at all. Second, an encoder pre-trained this way transfers better than standard masked pre-training on the same corpus when labeled data is scarce. This is the regime where Neural Edit Operations for Biological Sequences found that a built-in edit bias beats plain convolutions: its differentiable alignment layer (EINN) outperformed CNNs at 1–2% of the CB513 training data. EINN, however, pays for that bias at every forward pass and runs more than ten times slower than a CNN. We aim to put the bias into the weights during pre-training and pay nothing at inference.

The domain pre-training literature supports the idea that the corruption should match the domain. BioBART found that sentence permutation, a standard general-domain corruption, actually hurt biomedical tasks, because it destroys structure the domain relies on. Our reading is that corruption should imitate the variation the domain really has. For biology, that variation is indels, not shuffles or uniform token masking.

Research Program

For each sequence $x$ from a domain corpus, we sample an edited copy $\tilde{x}$ from a simple evolutionary edit model. The model has a per-position substitution rate, plus insertion and deletion rates with geometric lengths. While sampling, we record the alignment path $\pi$, which maps each position of $\tilde{x}$ to a position in $x$ or to a gap. A shared encoder embeds both sequences, and a cross-attention layer lets each token of $\tilde{x}$ attend over $x$. The objective is

$$\mathcal{L} = \mathcal{L}_{\text{MLM}}(\tilde{x}) + \lambda \sum_{i} \mathrm{JSD}\big(A_i \,\|\, P_i\big),$$

where $\mathcal{L}_{\text{MLM}}$ is the usual masked-token loss on the edited copy, $A_i$ is the cross-attention distribution of token $i$ of $\tilde{x}$ over $x$, $P_i$ is the target distribution derived from $\pi$, $\mathrm{JSD}$ is Jensen-Shannon divergence (the same divergence NWA uses), and $\lambda$ weights the alignment term.

One subtlety needs care. The true generative path and the optimal alignment can differ. A gap inside a homopolymer run, for example, can sit at any position in the run with equal score. We therefore spread $P_i$ evenly over positions that are equivalent in this way. We also inserted tokens to a learned "gap" slot, which plays the same role as the pruning weights in OTTC that let frames go unmatched without a blank token.

We need to be honest about which quadratic cost this removes. It eliminates the teacher's dynamic-programming table, which in NWA is a sequential per-pair computation that sits on top of the GPU work. It does not by itself remove the cost of the $n \times m$ attention map. For long sequences, we restrict cross-attention to a diagonal band. The band width is set by the indel budget we sampled, so it is known in advance rather than tuned by cross-validation as the banded kernels in Fast Global Alignment Kernels require. This brings memory to $O(n w)$ for band width $w$. The full-attention variant remains the reference at short lengths.

At inference the edited copy and cross-attention are dropped, and only the single-sequence encoder is kept, as in NWA.

The analysis has three parts. The first is a scaling study across sequence lengths, from the short pairs NWA was evaluated on (edit distance at most 40) up to multi-kilobase synthetic homologs. It asks whether the alignment benefit grows, plateaus or vanishes with length. The second is a mismatch study: we pre-train with an edit model whose rates differ from the evaluation data's true rates, to see how brittle the objective is. The third probes whether attention trained on synthetic edits recovers Needleman-Wunsch paths on real homologous pairs it never saw.

Baselines

The naive reference point is an encoder of the same architecture trained from random initialization on each downstream task, with no pre-training. Every pre-training gain is measured against it.

The strongest domain competitor is standard masked-token pre-training on the same corpus, as in TCR-BERT. TCR-BERT pre-trains on VDJdb and PIRD and then fine-tunes. It reports a mean area under the precision-recall curve (AUPRC) of 0.91 across 26 human antigens, and it outperformed general protein models such as ESM. We will rerun this recipe ourselves with identical architecture, corpus, tokens seen, optimizer and hyperparameter search budget, so the only difference is the objective. Beating it means a higher mean AUPRC on the same 26-antigen split. The gain must be clearest in reduced-label subsets, and it must not drop at full data.

The strongest alignment competitor is NWA (Needleman-Wunsch Attention), run with exact Needleman-Wunsch targets on its own benchmarks, DNA-Fountain and Qiita. It reports a 13% improvement in edit-distance approximation for Transformer encoders on DNA-Fountain and up to 3% lower root-mean-square error on Qiita. These are relative figures; no absolute values for these benchmarks were found on Lacuna. For long sequences, where the exact table is impractical, we give NWA a banded exact dynamic program as its teacher. This is the fairest cheap version of NWA, since banding is exact whenever the band covers the true edit drift. Our comparison is at equal wall-clock training budget, not equal steps, because the teacher's cost is the variable under study. Beating NWA means equal or better edit-distance root-mean-square error at lower training cost. On unpaired corpora, where NWA cannot run, the comparison is against the masked-token baseline.

For protein structure we use EINN (Neural Edit Operations for Biological Sequences) and its 12-block CNN, which reaches 71.5% accuracy on CB513 with full training data. EINN's advantage at 1–2% of the training data is reported only qualitatively; no reported number was found on Lacuna for EINN's low-data accuracy. We will reproduce both under the same data subsets.

Two ablations isolate the key ingredient. The first keeps the indel corruption but sets $\lambda = 0$. Any gain over it must come from the alignment target, not from the augmentation. The second keeps the alignment loss but corrupts with substitutions only, so the target is the trivial diagonal. This tests whether the benefit specifically comes from learning to track insertions and deletions.

Why not just…?

Why not just do masked-token pre-training with indel augmentation and skip the alignment target? This is the most serious threat. The encoder may learn indel robustness from reconstruction alone. The first ablation tests exactly this. If the $\lambda = 0$ variant matches the full objective on TCR binding, CB513 at low data, and edit-distance estimation, we will report that the free alignment signal adds nothing beyond augmentation. NWA's evidence points the other way: there, explicit attention targets beat the same encoders trained without them. But NWA trained on pairs, not on augmented single sequences, so the question is open and the ablation settles it.

Why not just scale NWA with a banded exact dynamic program? Banded Needleman-Wunsch is exact for pairs whose edit drift fits inside the band, so on paired data it is a real competitor. We run it as a baseline at matched wall-clock cost. The difference our method keeps either way is applicability. Banded NWA still needs related pairs, which domain corpora such as receptor repertoires generally do not supply. Our objective manufactures the pairs. If banded NWA beats us on DNA-Fountain and Qiita, that is informative: it would mean the optimal alignment is a better teacher than the generative path. We would then add a banded refinement of our synthetic pairs as a hybrid.

Why not just fine-tune a large general protein or DNA model? Within the file the evidence favors domain pre-training. TCR-BERT outperformed ESM on antigen binding. APEX shows the same pattern in a different modality, a domain-pre-trained model beating the best general model by 18% in mean absolute error. A study of parameter-efficient fine-tuning for code found domain pre-training a stronger predictor of downstream success than the adaptation method. We nonetheless include fine-tuned ESM on the TCR task as a reference, so the size of the gap is measured rather than assumed.

Proposed Experiments

The first experiment is the length-scaling study. We generate synthetic homologous DNA pairs at lengths from about 100 to several thousand bases, using the DNA-Fountain generation style. We compare our objective with full and banded attention against NWA with exact and banded teachers. We record edit-distance root-mean-square error and training wall-clock per epoch. We then test transfer to real Qiita microbiome pairs.

The second experiment is domain transfer under label scarcity. We pre-train on VDJdb and PIRD receptor sequences and evaluate antigen-binding AUPRC on the 26-antigen benchmark at 1%, 10% and 100% of labels. Separately, we pre-train on a protein corpus and evaluate CB513 secondary-structure accuracy at 1%, 2% and full CB6133 training data. EINN and the 12-block CNN serve as reference points there.

The third experiment is an alignment probe. On held-out real homologous pairs, we measure how often the argmax of the pre-trained cross-attention lands on the Needleman-Wunsch path. We compare this against the masked-token baseline's attention. This shows whether the edit-aware behavior learned from synthetic edits transfers to real biological variation.

Three outcomes would be decisive. The proposal succeeds if the full objective beats the $\lambda = 0$ ablation and the masked-token baseline in the low-label regimes, and matches NWA's edit-distance quality at a fraction of its training cost on long sequences. It fails if the augmentation-only ablation captures the whole gain. It is weakened if the edit-model mismatch study shows sharp degradation once pre-training indel rates diverge from the evaluation data.