Reading the Fading Memory: Detecting Long-Context Hallucinations in State Space Models from Their Own Decay Rates

State space models such as Mamba read a document the way a person skims: they keep a fixed-size running summary (the "hidden state") and let each piece of the summary fade at its own rate. This is why they scale linearly in document length. It also creates a specific way to fail. When a fact sits far back in a long filing or book chapter, the part of the summary that held it may have faded by the time the model answers, and the model fills the gap with something plausible. We propose to measure this fading directly. For every answer token, we compute how much of its evidence still traces back to the source document through slowly fading channels, as opposed to recent text or stored model knowledge. We then use that measure to detect and diagnose hallucinations in long documents at linear cost.

The core observation is that the tools needed here already exist but have not been combined. Mamba's computation can be rewritten exactly as an implicit attention matrix. Materializing that full matrix is quadratic in length, but a single row, meaning one answer token looking back over the whole document, costs only linear time. Separately, Mamba's channels split cleanly into slow-decaying "context-dependent" channels that carry factual information and fast-decaying local ones. Putting these together yields a per-token grounding signal that is both mechanistic and cheap. It also makes a falsifiable prediction that differs from what is known for Transformers: hallucination in a pure state space model should rise steadily with the distance between the evidence and the question, rather than following the U-shaped "lost in the middle" curve.

Core Claim

Our central claim has two parts. First, in pure selective state space models, hallucinations about document content are mostly caused by evidence decaying out of the slow channels, not by the model overriding the context with stored knowledge. Second, an "evidence retention score" can be computed alongside generation in $O(L)$ time. This score is the share of an answer token's implicit attention that reaches the document through slow channels. We expect it to detect such hallucinations better than generic hidden-state probes or token uncertainty, and to generalize better across document lengths and evidence positions.

Concretely, The Hidden Attention of Mamba Models shows that for channel $c$, the influence of token $j$ on token $i$ is

$$\tilde{\alpha}^{(c)}_{i,j} = C_i \Big(\prod_{k=j+1}^{i} \bar{A}^{(c)}_k\Big) \bar{B}_j,$$

where $\bar{A}_k$ is the input-dependent decay at step $k$, $\bar{B}_j$ is how strongly token $j$ is written into the state, and $C_i$ is how token $i$ reads the state. For a fixed answer position $i$, one pass with a running product of decays gives the entire row, at $O(L)$ cost per channel. That paper computed the full $L \times L$ matrix, which is exactly the quadratic cost it flagged as a limitation. Following Mamba Knockout for Unraveling Factual Information Flow, we partition channels into a slow set $\mathcal{S}$ and a fast set by their average decay. We then define the retention score at layer $\ell$ as

$$R^{(\ell)}_i = \frac{\sum_{c \in \mathcal{S}} \sum_{j \in D} |\tilde{\alpha}^{(c)}_{i,j}|}{\sum_{c} \sum_{j < i} |\tilde{\alpha}^{(c)}_{i,j}|},$$

where $D$ is the set of document token positions. The detector is a small logistic classifier over $R^{(\ell)}_i$ and two related quantities: the concentration (entropy) of the slow-channel row over $D$, and its total mass. It uses the layers in the 60–80% depth band, where the Knockout paper located factual transfer.

Relation to Prior Work

The papers closest to this proposal fall into three groups.

The first group supplies the method but studies only short prompts. Locating and Editing Factual Associations in Mamba uses causal tracing and rank-one editing to localize factual recall in Mamba-1-2.8B on COUNTERFACT triplets. Mamba Knockout finds a critical path at 60–80% depth and shows that slow-decay features carry factual flow, but only on 672 triplets that all models answer correctly. Neither paper looks at wrong answers, long inputs, or detection. We take their localization and channel split and apply them to failures in documents thousands of tokens long.

The second group motivates the proposal by establishing the gap. Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention proves that a fixed state cannot solve multi-query joint recall unless the state grows with the number of stored entities, and reports base Mamba-2 at 36.6% on that synthetic task. PHANTOM shows that Transformer detectors degrade with length and with evidence placed mid-document. The theory predicts that state models forget. Nobody has shown that this forgetting is what produces natural-language hallucinations, or that it can be read off at run time.

The third group sets the detection bar and constrains the design. HARP and ICR Probe probe Transformer hidden states. HARP projects onto a small subspace of the unembedding; ICR tracks whether feed-forward updates diverge from attention. ICR in particular depends on having separate attention and feed-forward blocks, which Mamba does not have. Towards Long Context Hallucination Detection is the strongest dedicated long-document detector. It chunks the text, encodes each chunk with a BERT-style encoder, and aggregates across chunks. HalluEntity shows that entropy-based detectors flag many correct entities, which rules out pure uncertainty as our signal. Our detector differs from all of these because its features have a causal reading, "the evidence faded," and cost linear time inside the model that is generating or reading the answer.

Research Program

The work has three stages.

The first stage is diagnosis. Using Mamba-1 and Mamba-2 checkpoints, we build controlled long-context probes by taking PHANTOM SEC filings and moving the gold evidence sentence to controlled distances from the question, from 500 to 30,000 tokens. We then measure answer accuracy and retention as functions of distance. To test whether a failure is caused by forgetting or by stored knowledge overriding the context, we use activation patching in the style of the Mamba editing paper. We restore slow-channel state from a run in which the evidence was placed close to the question, and check whether the hallucination turns into a correct answer. If patching slow channels repairs most failures and patching fast channels does not, the decay account is supported.

The second stage is detection. We train the small retention-feature classifier on one document type and one length range, then test on others. We run it in two modes. In reader mode, Mamba reads document, query, and a candidate answer, which is the setting of PHANTOM and BookSum. In generator mode, the score is computed on Mamba's own answers.

The third stage is an exploratory mitigation step, which we mark as secondary. When retention falls below a threshold, we take the document span with the highest slow-channel mass, append it just before the question, and regenerate. We compare this against appending a random span and against appending a lexically retrieved span. This comparison tells us whether the model's own faint attribution adds anything beyond retrieval.

Baselines

The naive reference points are length-normalized token likelihood and entropy computed from the same Mamba model. We also report the majority-class balanced accuracy. We expect these to have high recall but weak precision, because HalluEntity reports that valid entities also carry high entropy. The best entity-level uncertainty variant there, Focus, reaches AUROC 0.78 on Llama3-8B (HalluEntity). We will re-run the Focus-style keyword weighting on Mamba outputs so the comparison is within one model.

The strongest existing long-document detector on BookSum is Decompose-and-Aggregate from Towards Long Context Hallucination Detection. On its synthetic chapter-level split it reports 67.22% balanced accuracy, MCC 0.26, and 18.62 samples per second, against GPT-4o at 57.42% and 0.79 samples per second. We will reuse the same injected-hallucination split and report balanced accuracy, MCC, and throughput on matched hardware. Beating it means higher balanced accuracy and MCC without lower throughput. Matching its accuracy at clearly higher throughput on chapters longer than its training length would be a weaker but still useful result.

On PHANTOM, the strongest reported number is the PHANTOM-fine-tuned Qwen2.5-7B-Instruct at F1 0.933 on 10-K filings, up from 0.708 for the base model (PHANTOM). That detector was trained on full PHANTOM supervision. We will give our classifier the same training triplets and evaluate on the same 10-K split, and then evaluate on held-out form types and on lengths not seen in training. Outright beating 0.933 in-distribution is not expected, because our classifier has only a handful of parameters. The claim we would count as success is narrower: comparable F1 at a fraction of the memory, and a smaller drop than the fine-tuned model when moving to unseen form types and to mid-document evidence placements.

The most important competitor is a generic probe on the same Mamba model. This is a linear classifier on hidden states from the 60–80% depth band, plus a HARP-style variant that projects onto the low-variance singular directions of Mamba's unembedding. For HARP, no reported result on a state space model was found on Lacuna; its Transformer figure is 92.8% AUROC on TriviaQA with Qwen-2.5-7B-Instruct (HARP). These probes get the same labels, the same layers, and the same hyperparameter search budget as ours.

Two ablations isolate the key ingredients. The first removes the slow/fast channel split and uses aggregate implicit-attention mass over the document. The second keeps the channel split but does not restrict the numerator to document positions.

Why not just…?

Why not just use token entropy or likelihood? These are cheap and require no mechanistic machinery. However, HalluEntity shows that they flag valid, informative entities as hallucinations, and that hallucination rates for numbers (36.2%) and proper nouns (30.9%) are exactly where entropy is naturally high. Those token types dominate financial QA. The first baseline row measures this directly on Mamba. If entropy matches our score on PHANTOM, the mechanistic account adds nothing to detection, and we would report that.

Why not just train a linear probe on Mamba's hidden states? This is the real threat. A probe with enough data might learn the retention signal implicitly. The experiments settle it in two ways. First, the matched-budget comparison under shift: we train on 10-K filings of 5,000 tokens or fewer and test on 8-K and DEF 14A filings above 20,000 tokens, where a probe that has learned surface features of the training distribution should degrade more. Second, the patching result: only our feature has a causal interpretation that can be checked by restoring the state. If the generic probe matches us under shift, we will present the retention score as an explanation of what the probe learns, not as a better detector.

Why not just use a fine-tuned Transformer detector or GPT-4o? The PHANTOM fine-tuned detector is strong in-distribution. But PHANTOM itself documents Transformer degradation with length and mid-document evidence. GPT-4o and RefChecker are both slow (0.79 and 0.15 samples per second) and weak (57.42% and 53.99% balanced accuracy) on long BookSum chapters (Towards Long Context Hallucination Detection). Beyond that, a Transformer detector cannot say why a state space model failed, which is the diagnostic half of our contribution.

Why not just add sparse attention, as HAX does? HAX roughly doubles joint-recall accuracy, from 36.6% to 74.3% (Overcoming Long-Context Limitations of State-Space Models via Context-Dependent Sparse Attention). That is an architecture change, and it tells a user nothing about when a deployed pure model is guessing. The two approaches are complementary. We will also apply the retention score to a hybrid model where one is available, to test whether residual hallucinations there shift from forgetting toward stored knowledge overriding the context.

Proposed Experiments

The decisive diagnostic check is the distance sweep on PHANTOM documents. The decay account predicts that Mamba's hallucination rate rises roughly monotonically with evidence distance and tracks a falling $R_i$. PHANTOM's beginning/middle/end placements should then show the worst performance at the beginning, not the middle as reported for Transformers. A U-shaped curve for Mamba would weaken the account.

The second check is causal. Restoring slow-channel state should repair a large share of distance-induced errors, while restoring fast-channel state should repair few.

The third check is a negative control. On HalluEntity biographies, which have no source document, errors come from stored knowledge, so $R_i$ should carry little signal. If our score works well there, it is measuring something other than document retention.

A short sanity run on the 672 COUNTERFACT triplets from Mamba Knockout confirms that our row-wise computation reproduces the 60–80% depth band before we scale up.

The detection results then follow on BookSum synthetic chapters and PHANTOM, compared against the baselines above. We report AUROC, F1, balanced accuracy, MCC, throughput, and peak memory at 4K, 16K, and 30K tokens. Mitigation results are reported only as a secondary check, measured by answer accuracy after re-feeding the attributed span versus random and retrieved spans.