A Document-Length Linear Recurrent Memory for Noisy Domain-Specific NER

Named entity recognition (NER) is the task of finding and labeling names, organizations, drugs and similar spans in text. In specialized documents it often fails for a reason that has little to do with vocabulary. The clue that settles a mention's type sits paragraphs away from the mention. A French decree names a commune in its header and refers to it in abbreviated, OCR-garbled form forty lines later. A court judgment introduces the parties in a preamble and refers to them throughout. Standard transformer taggers read about 512 subword tokens at a time, so they cut these documents into windows and label each window alone. The existing fixes for cross-window consistency are span-level memories that match identical surface strings, or graphs built over spans proposed by a syntactic parser. Both are brittle on exactly the noisy, ungrammatical text where domain NER is hardest.

We propose a cheaper and more noise-tolerant mechanism. A domain encoder first labels the document window by window as usual. A bidirectional linear recurrent layer then runs once over the full concatenated sequence of token vectors. This is a recurrent network whose state update is linear, so the whole sequence can be computed in parallel in time proportional to its length. Its channels decay at many different rates, so some carry local boundary evidence and others carry document-wide evidence about what kind of thing a recurring name is. The output feeds a conditional random field (CRF), a decoding layer that enforces valid label sequences. The claim is that a soft, continuous document state recovers most of the consistency gains of span-memory and span-graph methods, without exact string matching, parsers or quadratic attention. We also expect its advantage over them to grow as character-level noise rises.

Core Claim and What Is New

Recurrent layers already appear in domain NER, but only at the sentence or record level. Information extraction from green channel textual records on expressways using hybrid deep learning stacks RoBERTa, a bidirectional GRU and a CRF on short toll-inspection logs and reaches 92.99% F1. It never asks the recurrence to carry information across a document. Named Entity Recognition With Parallel Recurrent Neural Networks shows that many small, decorrelated BiLSTMs beat one large one on CoNLL 2003 (91.48 F1), again within sentences. Both papers supply design ideas we reuse: recurrence on top of a pretrained encoder, and diversity across many small recurrent channels. Neither tests document memory.

Document-level NER work motivates the gap and also sets the bar. ScdNER stores span representations in a memory and retrieves identical surface mentions across the document. This cuts the inconsistency rate on CDR from 4.6% to 3.3% at 91.2% F1, but it depends on exact matches and on exhaustively enumerating candidate spans. Span Graph Transformer for Document-Level Named Entity Recognition connects parser-proposed spans in a graph and reports strong nested and flat results. Its graph is only as good as the constituency parser, which is a weak assumption for OCR administrative text.

On the recurrent side, Bidirectional Linear Recurrent Models for Sequence-Level Multisource Fusion (BLUR) supplies the method: diagonal linear recurrences with eigenvalues kept inside the unit circle, run as parallel scans in both directions. It reports up to 72x speedups over Informer. BLUR was evaluated only on forecasting and whole-sequence classification, never on token labeling. Our proposal is the first to test whether such a layer can carry entity-type evidence across thousands of tokens of a technical document, and whether it can do so under noise that breaks exact matching.

Mechanism

Let $x_1,\dots,x_N$ be the encoder's token vectors for an entire document of $N$ subwords, produced window by window. Each channel $j$ of the forward recurrence updates as

$$h_t^{(j)} = \lambda_j \, h_{t-1}^{(j)} + B_j x_t, \qquad |\lambda_j| < 1,$$

with a mirror-image backward pass. Here $\lambda_j$ is a learned per-channel decay and $B_j$ is a learned input projection. The value of $1/(1-|\lambda_j|)$ sets roughly how many tokens a channel remembers. We initialize the decays on a log-spaced grid, from a few tokens up to the typical document length. The intent echoes the finding in NeuralHydrology - Interpreting LSTMs in Hydrology that recurrent cells separate fast and slow dynamics on their own, and we impose that spread rather than hoping for it. Following Gregoric et al., we add an orthogonality penalty across channel groups so that long-memory channels do not collapse into copies of short-memory ones.

The two directions are fused with a gated linear unit, a small layer that multiplies one projection of its input by a learned gate. The fused output is added back to $x_t$ through a learned gate that starts near zero. This makes the model initially identical to the window-level tagger, and it must earn any change. A CRF decodes the final labels.

Research Program

We first establish whether the document state improves accuracy and consistency on three domains with long or multi-mention documents. The second strand is a controlled noise study. The third is a mechanistic analysis of what the long channels carry. For that we compute an analogue of the "time steps of influence" measure from the hydrology work: integrated-gradients attribution of each label to input tokens at increasing distances. We then check whether the attributed distant tokens are earlier mentions of the same entity. If accuracy rises but attribution stays local, the gain is coming from somewhere other than document memory, and we would report that.

Nonet's legal system is the main contrary evidence (Nonet at SemEval-2023 Task 6). Its hierarchical model fed window-level summary vectors into a sequential model and lost to simply keeping the last 512 tokens for judgment prediction. We read this as evidence against pooling each window to a single vector before the recurrence, which throws away token identity. It is not evidence against recurrence as such. Our layer runs over every token for that reason. The contradiction can still be tested: if our model also loses to window-level baselines on legal text, Nonet's result generalizes.

Baselines

The naive reference point is the window-level domain encoder with a per-token softmax head and no cross-window information. On AdminSet-NER this is the AdminBERT-NER setting. AdminSet and AdminBERT reports 80.11% F1 for AdminBERT-NER 16GB, and that is the number to beat on that benchmark. We add a CRF to the same encoder to separate the effect of constrained decoding from the effect of document memory. For CDR we use the same biomedical encoder in every system. The file does not name ScdNER's encoder, so we will match it as closely as its release allows.

The strongest existing method is ScdNER. On CDR we compare against its reported 91.2% F1 and 3.3% inconsistency rate, and we rerun its released or reimplemented consistency stage on our noised data. This competitor targets exactly the consistency problem we target, so it is the one most likely to make our layer redundant. We include the Span Graph Transformer where a parser can run; on CDR the file reports only a relative gain (+2.31% F1), not an absolute figure.

Two cheap context baselines stand in for what a practitioner would try first. One is overlapping sliding windows with extra context. The other is the multi-sentence concatenation training of Long Range Named Entity Recognition for Marathi Documents, which lifted long-range Marathi F1 from 72.64% to 85.46% without any memory. The third cheap baseline is post-hoc majority voting: every identical surface string in a document receives its most frequent predicted label.

The ablations that isolate our ingredient are three. First, a document-level BiGRU of matched parameter count in place of the linear layer, which tests whether linearity and parallelism cost accuracy. Second, the linear layer with all decays tied to a single short timescale, which removes long memory while keeping everything else. Third, the same model with the orthogonality penalty removed.

All systems share the training data, the encoder checkpoint and an equal hyperparameter search budget, measured in trials and GPU-hours. Results are reported as mean and standard deviation over five seeds. "Beating" a baseline means a mean F1 gain larger than two seed standard deviations, together with a lower document inconsistency rate.

Why not just…?

Why not just train on concatenated multi-sentence inputs, as for Marathi? This is a real threat on documents that fit in a few windows. Deshmukh et al. recovered more than 12 F1 points that way. Concatenation, however, cannot reach past the encoder's attention window, and AdminSet decrees and Indian judgments routinely exceed it. We will report gains bucketed by the distance between a mention and its nearest earlier co-referring mention. If concatenation matches us at every distance bucket, the recurrent layer is not needed.

Why not just vote labels across identical strings after decoding, or use ScdNER's exact-match memory? On clean biomedical abstracts this may work nearly as well; ScdNER's CDR numbers suggest consistency is largely a matching problem there. The open question is noise. Sebbag et al. report that OCR text and tokenizer over-segmentation disrupt names in administrative documents, which breaks exact matches. Our noise study injects character substitutions and segmentation errors into CDR at controlled rates and plots F1 against noise rate for voting, ScdNER and our layer. The proposal stands or falls on our curve being flatter.

Why not just use a document-level BiLSTM or BiGRU? It might match us in accuracy. Chen et al. found a BiGRU adequate on short logs, though nobody has tested gated recurrence at document length, where its single-state bottleneck is the known weakness. The matched-size BiGRU ablation answers the accuracy question directly. We also measure wall-clock training and inference time against document length. If the BiGRU ties on accuracy and the speed gap is small, we will say so.

Why not just prompt a large language model? The file argues against this for flat domain NER. On FiNER-ORD, fine-tuned RoBERTa (weighted F1 0.8792) beats few-shot Gemini-1.5 (0.8368) and GPT-4o (0.8321) (Financial Named Entity Recognition: How Far Can LLM Go?). Zero-shot Mixtral reaches 35.53% on AdminSet-NER. We include one few-shot model as a reference point only.

Proposed Experiments

AdminSet-NER is the primary benchmark: whole OCR administrative documents, 814 annotated texts, with the 80.11% AdminBERT-NER baseline. CDR is the consistency benchmark, measured against ScdNER's 91.2% F1 and 3.3% inconsistency rate, and it hosts the controlled noise study. SemEval-2023 Legal-NER tests long documents. The file reports only Nonet's 0.5532 F1, which ranked 15th, so we have no reported result on Lacuna for the top system on that task. We will therefore compare only against our own reproduced baselines there. FiNER-ORD, a set of financial news articles, serves as a check that the layer does not hurt on moderately long clean text.

Metrics are entity-level F1, the document inconsistency rate (the share of repeated surface forms given conflicting labels), F1 bucketed by co-reference distance, and throughput against document length.

Three outcomes would decide the question. The idea is supported if three things hold together: our model beats AdminBERT-NER plus CRF by more than seed noise on AdminSet-NER; it degrades more slowly than exact-match methods under injected noise on CDR; and attribution shows the long-decay channels drawing on earlier co-referring mentions. It is refuted if the tied-short-decay ablation performs as well, since then document memory is not the source of any gain. It is also refuted if concatenation training closes the gap at all distance buckets.