Learned Displacement Structure for Repeat-Aware DNA Sequence Models

Long-context DNA models such as HyenaDNA mix information along a sequence with very long convolutions. A convolution is a Toeplitz matrix, meaning the same weight is applied to every pair of positions at a given distance. That choice builds in one assumption: a pattern means the same thing wherever it sits. Repetitive DNA breaks this assumption in specific ways. Satellite arrays at centromeres repeat a unit whose length drifts and whose copies are grouped into higher-order blocks. Inverted repeats read the same pattern backwards on the opposite strand. Transposable elements are mosaics of shuffled modular pieces. We propose replacing the fixed convolution with matrices of low displacement rank (LDR). These are dense matrices that become low-rank after a simple "shift-and-subtract" operation, so they can be stored and multiplied almost as cheaply as a convolution. The shift itself is learned and allowed to vary by position, which lets the layer be "almost a convolution" under local warps of the coordinate system.

The core bet is that genomic repeats are shift-equivariant only up to position-dependent stretching and reversal, and that this is exactly the structure a learnable displacement operator encodes. If the bet is right, three things should follow. A drop-in LDR mixing layer should model satellite and structurally variable regions better than Hyena at matched parameters and compute. It should do no worse on standard regulatory benchmarks. Its learned operator parameters should line up with repeat boundaries and unit lengths, giving a readable repeat annotation for free. If the bet is wrong, a much simpler fix, adding a reverse-complement branch to Hyena, will match it, and the experiments are designed to show that cleanly.

Background and Gap

Displacement theory describes structured matrices by an equation $LA - AR = GH^\top$. Here $A$ is the $n \times n$ matrix of interest, $L$ and $R$ are fixed simple operators such as shifts, and $G, H \in \mathbb{R}^{n\times r}$ are thin "generator" matrices whose width $r$ is the displacement rank. With shift operators, Toeplitz matrices have $r \le 2$. Sums of Toeplitz and Hankel (anti-diagonal-constant) matrices have small $r$ with respect to symmetric shift operators. The matrix can therefore be stored in $O(nr)$ numbers and applied with fast Fourier transforms.

Three lines of prior work supply the method:

The skew-cyclic displacement work of Jiang et al. and Zheng et al. shows that a corner entry of $-1$ in the shift operator handles sign-flipped wraparound structure. In the subdiagonal family this is simply one value the learnable corner can take, so we include it as an initialization rather than a separate mechanism.

On the genomics side, HyenaDNA motivates the work and is the method we modify. Its implicit long convolutions, with filters produced by a small network of position, reach one million tokens at single-nucleotide resolution. Those filters are functions of relative distance only. Genomic language models: opportunities and challenges reports that Hyena- and Mamba-style models handle single-nucleotide variants but struggle with large insertions and inversions, which is the gap we target.

Three papers characterize the regime and supply real data:

We deliberately do not pursue the research file's suggestion of treating assembly graphs as displacement-structured matrices. De Bruijn and string graphs are sparse and irregular, so nothing in the file suggests their adjacency matrices have low displacement rank. Long reads and graph phasing already set a high bar there.

Core Claim

A sequence-mixing layer whose matrix has low displacement rank with respect to learned, position-varying subdiagonal operators can represent a family of structures that a fixed convolution cannot, even when the convolution is augmented with a reversed copy of itself. That family includes:

  • interactions that repeat with a slowly drifting period;
  • interactions that hold across a reversal;
  • interactions that hold across a locally rescaled stretch of sequence.

Each of these has small displacement rank under a suitable near-shift operator. On repeat-rich DNA, that representational advantage should become better likelihood and better structural-variant discrimination at the same parameter count and compute. The learned subdiagonal values should act as a per-position "local clock" that marks unit boundaries.

Research Program

We build the LDR mixer by taking the HyenaDNA architecture and replacing each long implicit convolution with an LDR-SD matrix, keeping Hyena's gating and short filters. Storing explicit generators of length $n$ for 100k-token contexts would inflate parameters far beyond Hyena's. To avoid this, the generators $G, H$ and the subdiagonal entries of $L, R$ are themselves produced by small networks of position, in the same implicit style Hyena uses for its filters. The new parameters relative to Hyena are then only these small networks. Displacement rank $r$ ranges over 1 to 8.

The subdiagonal entries are parameterized with magnitude at most one and initialized at exactly one, so the layer starts as a Hyena-like convolution. The corner entry is free, which allows both cyclic (+1) and skew-cyclic (−1) wraparound.

Numerical stability is a known hazard. Faster Sparse Matrix Inversion and Rank Computation in Finite Fields notes severe floating-point instability for displacement-based inversion, and the Zheng et al. stability guarantee covers only well-conditioned matrices. We need only matrix–vector products and their gradients, never inversion, which avoids the worst failure mode. Long products of subdiagonal entries can still vanish or explode over $10^5$ steps. The bounded parameterization and float32 accumulation are our defense. We will measure relative error of the fast multiply against a dense reference at moderate $n$ before any training.

We then construct a repeat-structure evaluation suite, because the file contains no existing repeat-specific benchmark. It has a synthetic tier and a real tier.

The synthetic tier generates sequences on a random background containing one of three kinds of planted structure:

  • tandem arrays whose unit length drifts, or whose copies are grouped into higher-order blocks;
  • inverted copies at controlled distances;
  • mosaic elements assembled by shuffling and inverting domains drawn from a domain library, in the spirit of Zhi et al.

Generator parameters are known exactly, so we can measure masked-token likelihood inside planted regions, copy-number regression, and inversion-breakpoint localization. Difficulty can be swept along one axis, for example period drift rate, while the others stay fixed.

The real tier uses the haplotype-resolved human assemblies and centromeres of Logsdon et al. and the ape assemblies of Yoo et al. Models are trained on hg38 plus a subset of these haplotypes, with whole chromosomes and whole individuals held out. Evaluation covers three tasks:

  • held-out perplexity stratified by region type (centromeric satellite, segmental duplication, other repeat, unique);
  • classification of windows containing an assembly-derived inversion or duplication against matched windows without one;
  • prediction of higher-order repeat unit length in held-out centromeres.

Labels for the last task come from the assemblies' own annotations. Where we derive labels from self-similarity, we report results separately and treat them with caution.

Finally, we analyze the learned operators. For each position we extract the learned subdiagonal profile and test whether its change points coincide with annotated repeat-unit and domain boundaries. We report precision and recall at a fixed tolerance. We also report the effective displacement rank actually used, measured by the singular-value decay of the generators, across region types.

Baselines

The naive reference point is a k-mer frequency model of matched order, together with a small Transformer using windowed attention at the longest context it can afford. Together they show how much any long-range mixing helps on each task. Beating them is necessary but uninteresting.

The strongest existing method is HyenaDNA itself. The file reports its results only on regulatory benchmarks: state of the art on 12 of 18 Nucleotide Transformer datasets, 7 of 8 GenomicBenchmarks datasets, and gains of +10 to +20 accuracy points on enhancer identification, with 1.6M parameters outperforming the 2.5B-parameter Nucleotide Transformer (HyenaDNA). We found no reported result on Lacuna for HyenaDNA or any other model on centromeric perplexity, inversion-window classification, or higher-order repeat period prediction. Those numbers must therefore be produced in this project.

We will retrain HyenaDNA from the released recipe on exactly our training data, contexts and token budget. It gets the same hyperparameter search budget as our model (equal numbers of trials over learning rate, width and depth). Model sizes are matched in parameters and in measured GPU-hours rather than FLOP estimates, because LDR-SD has worse constants than a plain FFT convolution. On the regulatory benchmarks we also check that our retrained HyenaDNA roughly reproduces the published figures, so that we are not beating a weakened baseline.

The key ablations isolate the learnable operator. "Hyena+RC" adds a branch that applies the same implicit convolution to the reverse-complemented input, which yields a Toeplitz-plus-Hankel mixing matrix. "Fixed-LDR" uses our layer with $L, R$ frozen at shift operators, so the generators can still produce Toeplitz-plus-Hankel-like matrices but cannot warp the clock. "LDR-SD" is the full model, and a variant with the corner frozen to +1 tests whether skew-cyclic wraparound matters.

Beating the strongest baseline means two things, both at matched compute and over at least three seeds:

  • LDR-SD improves over both HyenaDNA and Hyena+RC on centromeric and segmental-duplication perplexity and on inversion-window classification, with non-overlapping seed intervals.
  • It loses no more than one seed-standard-deviation on GenomicBenchmarks.

Why not just…?

Why not just add a reverse-complement branch to Hyena? This is the real threat. Many inversion effects are captured by a Toeplitz-plus-Hankel matrix, which Hyena+RC already provides. If Hyena+RC matches LDR-SD on inversion tasks, we will report that the needed bias is reversal symmetry alone. The proposal's claim then narrows to drifting-period and mosaic structure. The synthetic period-drift sweep settles this. A fixed convolution, reversed or not, applies one distance-to-weight map everywhere, so its advantage should shrink as drift grows, while a learned local clock should not. If LDR-SD shows no advantage there either, the idea has failed.

Why not just train a bigger HyenaDNA or fine-tune it on repeat-rich data? Scale and data address capacity, not the coordinate assumption built into convolution. Our compute-matched comparison speaks only to equal budgets. We will add one HyenaDNA run at twice our compute, fine-tuned with repeat-region upweighting. If that closes the gap, the structured layer is a convenience rather than a necessity, and we will say so.

Why not just use self-similarity or tandem-repeat-finding heuristics for annotation? For pure annotation, alignment-based self-identity tools are strong and need no model. The proposal does not compete with them as an annotator. Its claim is that the language model's internal structure aligns with repeat units, which matters for downstream use such as variant effect prediction inside repeats. We use those heuristics as a source of reference labels, not as a rival on modeling tasks.

Why not use low-rank or LoRA-style compression instead? Low-rank matrices cannot represent full-rank shift-like mixing. Sindhwani et al. and Thomas et al. both show LDR beating low-rank at equal parameters, so we do not spend budget on this comparison beyond a single sanity run.

Decisive Checks

The project is worth continuing only if four checks pass. First, the fast LDR-SD multiply stays within 1e-4 relative error of a dense reference at 16k tokens after training. Second, LDR-SD beats Fixed-LDR, which isolates the learnable operator, on the synthetic drifting-period tier by a margin that grows with drift rate. Third, the same ordering appears on held-out real centromeres and segmental duplications, with HyenaDNA and Hyena+RC trained under identical budgets. Fourth, regulatory-benchmark accuracy is preserved. The interpretability analysis, where subdiagonal change points match annotated unit boundaries above a shuffled-position control, is the supporting result that would make the layer useful beyond a perplexity gain.