Vision state space models such as Vision Mamba process an image by flattening its patches into a sequence and reading them one at a time. A small running memory, the "hidden state," summarizes everything seen so far, so cost grows linearly with image size. That linear cost is why these models are attractive for high-resolution field cameras on edge hardware. The same design has a weakness. A patch of solar glare, a raindrop on the lens, or a deep canopy shadow is written into the running memory like any other patch, and it then distorts the model's view of every clean patch that comes after it along the scan. The obvious fix is to let the model refuse to write corrupted patches into memory. Current selective state space models cannot do this well, because the gate that decides how much to write sees only the current patch and a few neighbours. It never compares that patch with what the memory already holds.
We propose to measure this "state poisoning" directly on real field hazards, and then to fix it with a change that adds no test-time training. The memory gets a cheap one-step prediction of the next patch. When the actual patch is very surprising relative to that prediction, the gate shrinks how much the patch is written into long-range memory. The patch's own output is left alone. The goal is a lab-trained model that degrades gracefully in the field, keeps constant per-patch inference cost, and needs no backpropagation at deployment.
Core Claim
Localized physical hazards cause measurable damage that is directional. Features of clean patches downstream of a hazard along the scan change much more than features of patches upstream. A gate that can see only the current token cannot learn a general rule for blocking this damage.
The fix is a state-conditioned write gate, trained once with localized hazard augmentation. It should shrink the downstream damage and improve lab-to-field transfer to hazard types it never saw during training. Hazard augmentation alone, given the same data, should not achieve the same. The second half of the claim is the one that could fail, and the experiments are built around it.
Relation to Prior Work
The poisoning problem has already been named. TRUST identifies "hidden state poisoning" and fixed-scan bias in VMamba and counters them at test time. It runs several scan orders, keeps the lowest-entropy ones, updates the scan parameters by gradient descent on pseudo-labels, and averages the updated weights. It reports 56.1% mean accuracy on ImageNet-C against 41.7% for Tent. That makes it the strongest reference for corruption robustness. TRUST, however, pays with backward passes per test batch and roughly 1.7x memory for eight traversals, and it was only tested on synthetic corruptions. We aim for a fully feedforward model and test it on physical hazards.
Two other papers mute parts of the state, but along a different axis. DGMamba uses the input-dependent discretization (the step size that sets how fast memory decays and how strongly the current patch is written) to suppress states it judges to be background. It also shuffles Grad-CAM-selected context patches. UIS-Mamba mutes background patches found by a Normalized Cut, with a suppression factor φ to which segmentation proved fragile.
Both separate object from background semantically. Our gate separates reliable from unreliable observations, and glare on a leaf is foreground. DGMamba's shuffling also destroys the spatial continuity that dense lesion segmentation needs. These two papers motivate the idea that state writes should be controlled. The UIS-Mamba result also constrains us: hard muting is unstable, so our gate is soft and acts on only part of the state.
DGFamba treats domain shift as a global change of style statistics, regularized with an optimal-transport loss. It does not address localized corruption. ConMamba is the closest application paper. It reaches 94.29% on PlantDoc with a bidirectional Mamba encoder and contrastive losses, but uses fixed scans and does not analyze how hazards propagate.
CodeMamba models background clutter as a manifold and flags departures from it as targets. It is the nearest relative of "surprise," but uses surprise as the detection output rather than as a control on memory writes.
On the theory side, Theoretical Foundations of Deep Selective State-Space Models frames selective layers as input-controlled differential equations, in which the gates are driven by the input path. This supplies our reading that a token-local gate is the wrong place to detect "inconsistent with context." Its result that diagonal layers need depth to mix channels also explains why later layers cannot easily undo a poisoned state.
Mechanism
Consider one selective scan with hidden state $h_t$, patch embedding $x_t$, and learned step size $\Delta(x_t)$. We add a small learned linear map $P$ that predicts the next embedding from the previous state. From it we compute a surprise score $e_t = \lVert x_t - P h_{t-1} \rVert / s_t$, where $s_t$ is a running scale of recent errors. The surprise score modulates the step size on a designated subset $\mathcal{G}$ of state channels:
$$ \Delta'_t = \Delta(x_t)\,\sigma\!\left(a - b\,e_t\right) \ \text{on channels in } \mathcal{G}, \qquad \Delta'_t = \Delta(x_t) \ \text{elsewhere}, $$
Here $\sigma$ is the logistic function, and $a$ and $b$ are learned scalars.
When $\Delta'_t \to 0$, the update becomes $h_t \approx h_{t-1}$: memory is held across the surprising patch and integration resumes afterwards. The patch still reaches its own output through the skip term $D x_t$ and the ungated channels. This matters because small lesions are also surprising. We want them to be visible locally, and to influence downstream patches only through channels the model chooses.
The cost is one extra matrix-vector product per step. Constant cost per patch is therefore preserved. One complication is that the gate now depends on $h_{t-1}$, which breaks the parallel associative scan in its plain form. To keep training parallel, we compute $P h_{t-1}$ from a first, ungated pass and then run the gated pass. This doubles scan cost only in the layers that carry the gate. We will report wall-clock cost for this choice.
Research Program
Measurement. First we build a poisoning probe. We paste localized hazards into clean images and compare every block's features at patches outside the hazard mask against the same image without the hazard. The pasted hazards are rendered glare blobs, saturated regions, rain streaks, motion-blur bands, and shadow penumbras. We plot the feature deviation against signed distance along each scan direction.
Poisoning predicts an asymmetry: downstream deviation is larger than upstream deviation at equal spatial distance. A ConvNeXt run through the same probe serves as the symmetric control. If source VMamba shows no asymmetry beyond the CNN control, the premise fails and we stop.
We repeat the probe on real hazards in WildDash. There we mark overexposed regions by pixel saturation and use the clean remainder of the frame as the measurement region. Because no clean twin of a real frame exists, we compare predictions on the clean region with and without masking the hazard input.
Training. Second, we train gated and ungated VMamba backbones under an identical recipe, with the same hazard augmentation. Third, we test transfer to hazard types and domains that were held out during training.
Baselines
The naive reference is the source VMamba backbone with standard augmentation and no hazard handling. Every gain is measured against it.
The most dangerous competitor is the same VMamba trained with exactly our localized hazard augmentation, using the same images, epochs, and hyperparameter search budget (an equal number of trials per arm). If augmentation alone gives equal held-out-hazard accuracy and an equal poisoning asymmetry, our gate adds nothing.
For dense field segmentation, the strongest reported architecture is ConvNeXt-L at 46.24% mIoU on PlantSeg, reported in the PlantSeg paper. That paper also reports MobileNetV3 at 10.22% mIoU as the lightweight floor. We retrain both under our protocol. We count a win as matching or exceeding ConvNeXt-L's mIoU at lower edge latency, and as exceeding the augmented VMamba arm.
For corruption robustness, the strongest method is TRUST: 56.1% ImageNet-C mean accuracy, versus 41.7% for Tent, as reported in TRUST. We run TRUST on our backbone in its published configuration. We report accuracy together with per-frame latency and memory, since our claim is a better accuracy–latency trade-off, not higher peak accuracy. Our feedforward model may also be combined with TRUST, to test whether the gains add up.
DGMamba's hidden-state suppression module (without patch shuffling, which breaks dense prediction) is run as the semantic-muting alternative.
Two ablations isolate the key ingredient. The first applies the gate to all channels instead of a subset, which probes the UIS-Mamba-style instability. The second keeps the extra prediction head $P$ but conditions the gate on $x_t$ only. This controls for added parameters and tests whether conditioning on the state is what matters.
For driving, the reference numbers come from WildDash: DRN-D38 trained on Cityscapes reaches 16.5 mIoU, trained on Mapillary Vistas reaches 30.7, and scores 7.2 on negative test cases. These are older CNN numbers and are context, not targets. For edge latency on a Jetson-class device, no reported result was found on Lacuna. TomFormer notes high latency on a Jetson Nano without giving a figure, so we will measure all arms ourselves.
Why not just…?
Why not just train with localized hazard augmentation? This is the real threat, and it is why every arm receives identical augmentation. Our argument for the gate is that augmentation teaches output invariance to the hazard appearances it saw. Later layers can achieve that while the state stays poisoned, and the fix may not carry over to new hazards. A state-conditioned gate implements a generic rule: do not write what contradicts context. Two measurements settle this: the held-out-hazard split (train on glare and blur, test on rain and shadow, and the reverse) and the asymmetry probe. If augmentation matches the gate on both, we will report that augmentation suffices and that the gate is unnecessary.
Why not just use a convolutional or hybrid backbone, which has no recurrent state to poison? ConvNeXt-L does lead on PlantSeg at 46.24% mIoU. But TomFormer reports that a small hybrid still had high latency on a Jetson Nano. UIS-Mamba beat a 700M-parameter model with 115M parameters. The question is therefore the accuracy-per-latency frontier on edge hardware. We plot that frontier for all backbones, and if ConvNeXt dominates it, the case for state space models in this setting weakens regardless of our gate.
Why not just adapt at test time with TRUST? TRUST gives large corruption gains, but it needs backward passes, multiple traversals, and a batch of test data. That fits poorly with 30-plus frames per second of streaming robot video, and continuous adaptation on corrupted streams risks collapse. We run TRUST on per-frame streams from the hazard-rich WildDash cases and report accuracy against latency. If TRUST's accuracy margin survives at a deployable latency, we will say so and position the gate as complementary.
Why not just mute background states as DGMamba does? Semantic muting suppresses what is not the object. Glare and shadow fall on the object itself, so a background mask leaves them in. The comparison on PlantSeg images with hazards on the leaf, scored by mIoU restricted to hazard-overlapping lesions, tests this directly.
Proposed Experiments
The lab-to-field transfer test trains classifiers on PlantVillage and tests on PlantDoc. It also trains segmenters on PlantSeg and tests on KUTomaDATA greenhouse images through box-level detection mAP, where TomFormer reports 87%. We know of no reported Lacuna number for PlantVillage-to-PlantDoc cross-domain transfer, so all arms are run by us.
For driving, we train on Cityscapes and test on WildDash's public validation cases, broken down by hazard cluster. We also report ImageNet-C accuracy per corruption, to connect to TRUST's numbers. Each experiment uses three seeds, and we report means with spread.
The decisive checks are three. First, the source model must show downstream-greater-than-upstream deviation on real WildDash glare that the CNN control lacks. Second, the gate must cut that asymmetry relative to the augmentation-only arm. Third, the gated model must beat the augmentation-only arm on held-out hazard types while losing no small-lesion IoU on PlantSeg (lesions under 1% of image area). The last condition is the risk that surprise-gating mutes real disease spots.
All three must hold for the claim to stand. Failing the first kills the premise, failing the second kills the mechanism, and failing the third kills its usefulness for agriculture.
References
- TRUST: Test-Time Refinement using Uncertainty-Guided SSM Traverses
- DGMamba: Domain Generalization via Generalized State Space Model
- UIS-Mamba: Exploring Mamba for Underwater Instance Segmentation via Dynamic Tree Scan and Hidden State Weaken
- DGFamba: Learning Flow Factorized State Space for Visual Domain Generalization
- ConMamba: Contrastive vision Mamba for plant disease detection
- CodeMamba: Shifting from Target Semantics to Self-Supervised Background Manifold Learning for Singularity Detection in Infrared Sequences
- Theoretical Foundations of Deep Selective State-Space Models
- A Large-Scale In-the-wild Dataset for Plant Disease Segmentation
- WildDash - Creating Hazard-Aware Benchmarks
- Early and Accurate Detection of Tomato Leaf Diseases Using TomFormer