Robust Tickets as a Label-Free Fix for Compression Bias in Genomic lncRNA Classifiers

Long non-coding RNAs (lncRNAs) are RNA transcripts longer than 200 nucleotides that do not code for protein but regulate development and disease. Finding them in newly sequenced genomes is increasingly done by large DNA language models, which are Transformers pre-trained on genomes split into short nucleotide "words." These models are expensive. To run them across whole assemblies of non-model plants, people will compress them by pruning weights. Work on compression in vision shows that pruning does not cost every group equally: the rare, hard groups lose the most. In a genome, the rare groups are exactly the transcripts people most want to find. These include lncRNAs from under-sampled species, transcripts from fragmented assemblies, and short, GC-unusual, or near-coding transcripts.

This note proposes a direct test of a single hypothesis. Pruning a DNA language model makes it lean harder on a few cheap sequence cues, such as the poly-A/poly-T flanking tracts that the uncompressed model already relies on. Those cues work for common strata and fail for rare ones. The same reliance also makes predictions flip after a handful of base edits. If that is true, two things follow. First, a model's fragility to small, biologically plausible edits can be measured on an unlabeled genome, and it should flag in advance which strata pruning will hurt. Second, searching for the pruned subnetwork under a worst-case edit objective should shrink the per-stratum damage without any group labels. The second point matters because group labels for new species do not exist, and because label-based fairness constraints are known to stop working off the training set.

Core Claim

The core claim has three parts. Pruning a genomic lncRNA classifier produces a measurable excess accuracy gap on rare biological strata. Label-free few-edit fragility predicts the size of that gap per stratum. A robust "ticket" (a sparse subnetwork found inside the pre-trained model) chosen by adversarial masked-nucleotide modeling on the target genome closes more of the held-out-species gap than an explicit group-constrained method trained on source-species labels.

The parts play different roles. The lncRNA pipeline is the application regime and supplies the strata. Compression-induced bias is the failure being measured. Robust subnetwork search is the proposed remedy.

Each part is novel relative to the file. None of the cited lncRNA models has been compressed. None of the robust-ticket papers measures per-group damage. The group-fairness pruning methods use group labels and have never touched sequence models.

Background and Positioning

The application setting comes from DNABERT-based explainable lncRNA identification in plant genome assemblies. That paper fine-tunes a human-pretrained DNA Transformer to call lncRNAs directly from genomic sequence plus 500 bp flanks across seven plant species. It motivates this proposal in two ways. Its accuracy falls from 83.4% on Zea mays to 57.9% on the less complete Brassica rapa assembly and to 61–63% across species, so strata are already uneven in the dense model. Its attribution analysis names poly-A/poly-T flanking tracts as primary cues, which is the candidate shortcut. That paper never compresses the model; we do.

Feature-based pipelines constrain the design rather than supply it. IIMLP builds 35 entropy and information-theoretic features and reaches 99.79% AUC on balanced, redundancy-filtered human GRCh37 transcripts. LncPred-IEL/ANEL combines handcrafted and k-mer features with attention and beats CPAT and CPC2, but is tested only at mild 1:2 and 1:3 imbalance. Both methods operate on assembled transcripts with known ORFs, so they do not address the assembly-level setting. They are, however, the honest reference for "small model, no compression needed," and we keep them as baselines.

Compression-bias work supplies the measurement and the main competitor. Balancing Act (CEAG) defines the excess accuracy gap (EAG), meaning how much more a group loses than the average. It bounds this gap with a constrained min-max fine-tune. It also reports that the gap returns on test data, and that is the opening this proposal targets. LTAP scores weights with several criteria, reweighted by class accuracy, and raises CIFAR-100-LT tail accuracy from 23.8% to 34.1%. Fairness-aware adversarial pruning learns masks against a discriminator that predicts demographic attributes. All three need group or class labels at pruning time, and all are vision-only.

Robust-ticket work supplies the method. ORTicket finds one structured BERT subnetwork using adversarial masked language modeling. That subnetwork transfers robustness to downstream tasks without task-specific adversarial training: IMDb accuracy under TextFooler goes from 28.4% to 61.6%. Double-Win tickets show that robustly pre-trained models contain sparse subnetworks whose flat loss basins transfer with 1% downstream data. Neither paper asks whether its subnetwork drops rare classes. We reuse ORTicket's label-free adversarial masked-modeling search, adapted to nucleotides. We then ask the question both papers skip.

The evaluation discipline comes from Adversarial Pruning: A Survey and Benchmark. It found that robustness was overstated by up to 6.37% under weak attacks, and that unstructured pruning keeps robustness far better than structured pruning. We therefore use strong, multi-restart discrete attacks and report both pruning granularities.

Research Program

Strata. We build stratified lncRNA benchmarks in two settings. The first is the seven-species plant assembly suite from the DNABERT study. The second is human and mouse lncRNA versus protein-coding sets from GENCODE, drawing on the GENCODE 2025 annotations together with the CPPred imbalanced sets. Each example is tagged with:

  • species;
  • assembly completeness;
  • length bin;
  • GC bin;
  • genomic context (antisense, intergenic, or overlapping a coding locus);
  • presence of a short open reading frame (a near-coding hard case);
  • for human and mouse, expression level where it is annotated.

We also build imbalanced versions at 1:10 and 1:20 lncRNA to coding, well beyond the 1:3 ratio tested so far.

Perturbations. We define a biologically bounded edit set $\Delta_k$: all sequences reachable from input $x$ by at most $k$ substitutions, single-base indels, or short N-masked runs. These mimic sequencing errors and assembly gaps. A sequence's fragility is the smallest $k$ at which a strong search, run with greedy and beam search and multiple restarts, flips the prediction. Fragility needs no label beyond the model's own prediction, so it can be computed on any unlabeled genome.

Measurement study. We prune a DNABERT-style lncRNA classifier with magnitude pruning, both unstructured and head- or neuron-structured, at 50%, 70%, and 90% sparsity. For every stratum we record EAG and fragility. We then test whether stratum-level dense fragility rank-predicts stratum-level EAG after pruning. A separate shortcut test checks the mechanism: we shuffle or replace poly-A/T flank tracts while preserving composition, and measure whether prediction changes grow with sparsity and are concentrated in the high-EAG strata.

Remedy. We find a mask $m$ over pre-trained weights $\theta$ by solving

$$ \min_{m}\; \mathbb{E}_{x \sim G}\, \max_{\delta \in \Delta_k} \mathcal{L}_{\text{MNM}}\big(f_{m \odot \theta}(x \oplus \delta)\big). $$

Here $G$ is unlabeled sequence from the target genome, $\mathcal{L}_{\text{MNM}}$ is the masked-nucleotide prediction loss, $x \oplus \delta$ applies the edit $\delta$, and $\odot$ is elementwise masking. The inner maximization is done in embedding space during search, following ORTicket, and with discrete edits at evaluation. The ticket is then fine-tuned on source-species labels only. No group labels enter at any point.

Baselines

Dense model. The naive reference is the uncompressed DNABERT-style classifier fine-tuned on source species. The cited paper reports 83.4% accuracy on Z. mays, 57.9% on B. rapa, about 73.5% median across species, and 61–63% across species. Plain magnitude pruning at matched sparsity is the second naive reference. It defines the compression damage that every method must undo. Beating it means a lower worst-stratum EAG at equal sparsity and equal clean accuracy within one point.

CEAG with species and stratum labels. The strongest competitor is CEAG run on the pruned model with the source species' strata as groups, using its replay buffer and an $\epsilon$ tolerance tuned on a source-species validation split. It is the strongest because it directly optimizes the quantity we measure, using information our method withholds. There is no reported result on Lacuna for CEAG or any group-constrained pruning on genomic or lncRNA data. Its only reported effect in the file is on vision: removing a roughly 10% excess drop on UTKFace training data.

LTAP and the fairness-aware adversarial pruner are run the same way. They get identical sparsity targets, the same pre-trained checkpoint, the same fine-tuning steps, and the same hyperparameter search budget of 20 trials per method.

Our method must win where it counts. It must match CEAG's worst-stratum EAG on source-species test data within a pre-set margin of two points. It must also beat CEAG's worst-stratum accuracy on held-out species, where CEAG's own paper predicts its constraint will weaken.

Feature-classifier reference. We train IIMLP-style entropy features and LncPred-style feature models on the same genomic windows, and report their per-stratum accuracy as the "no deep model at all" point. For calibration, IIMLP reports 99.79% AUC on balanced transcript-level GRCh37. That is a different input setting, and we will report both settings.

Ablations. Three ablations isolate the key ingredient:

  • a random-edit ticket, with the same search but random rather than worst-case edits;
  • a clean ticket, with adversarial-free masked-modeling search, as in plain lottery-ticket pruning;
  • a source-genome ticket, searched on source rather than target unlabeled sequence.

Why not just…?

Why not just use CEAG with whatever labels exist? This is the real threat. On source species, CEAG may simply win, since it optimizes the measured gap directly. Its own paper documents that the gap reopens on unseen data, and new species are the extreme case of unseen data. The held-out-species comparison settles this. If CEAG still wins off-distribution, the claim fails and we report it.

Why not just add random mutation noise during fine-tuning? Random noise is cheaper than worst-case search, and it may cover realistic sequencing error. The random-edit ticket ablation tests exactly this. If it matches the adversarial ticket on worst-stratum held-out accuracy, the conclusion becomes "noise-aware ticket search suffices." That is still useful, and simpler. The adversarial maximization should be dropped in that case.

Why not just use a small feature model and skip compression entirely? Entropy and k-mer models are already small and score extremely well on balanced transcript benchmarks. They also outperform older tools, per LncPred. But they assume assembled transcripts, and the assembly-level setting is where the dense language model's flank motifs matter. We run them on identical genomic windows. If they match the pruned language model per stratum, the efficiency motivation disappears, and the note's contribution narrows to the measurement study.

Why not just distill into a small network? Fairness-aware adversarial pruning reports that a sparse model trained from scratch reaches 0.5071 equalized-odds difference versus 0.2149 for extracted subnetworks on CelebA. ORTicket reports collapse when its subnetwork is re-initialized. Neither is genomic evidence. We include a distilled small CNN with matched parameter count as a baseline rather than assume it loses.

Proposed Experiments

The first decisive check is whether compression bias exists here at all. At 90% unstructured sparsity, magnitude pruning should raise worst-stratum EAG well above the average drop. We expect the largest rises in B. rapa-like fragmented assemblies, short-ORF transcripts, and GC-extreme bins. If no stratum shows an excess gap, the study stops at a negative result.

The second check is whether dense-model fragility rank-correlates with post-pruning EAG across strata (Spearman, with bootstrap intervals over strata). We also test whether flank-shuffling sensitivity grows with sparsity, which would confirm the shortcut mechanism.

The third check is leave-one-species-out evaluation on the seven-plant suite. Using matched sparsity and budgets, we compare the robust ticket against CEAG, LTAP, plain pruning, the three ablations, the feature classifiers, and distillation. The primary metric is worst-stratum accuracy on the held-out species. Secondary metrics are mean accuracy, EAG, and accuracy under $k \in \{1,2,4\}$ worst-case edits, evaluated with the strong attack protocol.

We repeat everything at 1:10 and 1:20 imbalance on human and mouse GENCODE data. We report structured and unstructured masks separately, because the adversarial-pruning benchmark warns that structured pruning may carry the robustness penalty that would sink the remedy at hardware-friendly granularity.