Frozen Depth Reservoirs: Untrained Inter-Slice Memory for 3D Medical Segmentation

Many hospitals and labs segment 3D scans (CT, MRI, electron microscopy) by running a strong 2D image model on each slice independently. The 2D model is cheap and benefits from large-scale pretraining, but it never sees how a structure continues from one slice to the next. An organ can appear on one slice and vanish on the next, and boundaries come out jagged. The usual fixes are expensive. One is a full 3D network. Another is a recurrent network trained to carry memory across slices, which requires backpropagating gradients through the entire stack of slices.

We propose a third option. A fixed, randomly wired dynamical system, which is never trained, sweeps through the stack of frozen 2D features in both directions. It steps according to the physical distance in millimetres between slices rather than the slice index. Only a small per-slice output head is trained, and no gradient ever crosses the depth axis. If this works, a single cheap model would supply inter-slice context and could be trained on a few dozen scans. Because its memory is defined in millimetres, it should also keep working when a scanner uses thicker or thinner slices than the training data.

Core Claim

The idea borrows from reservoir computing. In this approach, a sequence is fed into a large recurrent network whose weights are random and frozen (the "reservoir"), and only a readout layer on top of its states is learned. Our claim is that inter-slice context for 3D segmentation can be supplied this way: a fixed bidirectional reservoir integrates frozen 2D encoder features over physical depth, and a trained per-slice readout uses the result.

The claim has three testable parts:

  1. Accuracy. This recovers most of the accuracy that trained recurrent slice models gain over plain 2D inference.
  2. Cost. It does so at a fraction of the training memory, because reservoir states are computed once, cached, and reused.
  3. Robustness to slice thickness. Because the reservoir's step size is tied to physical slice spacing, the same readout transfers across slice thicknesses. Methods whose context is defined in slice-index units (2.5D stacking, fixed-size 3D kernels, discrete-step recurrent networks) should degrade under this shift.

The third part is what distinguishes the proposal from simply "an echo state network on medical images," and it is where we expect the clearest win.

Where This Sits

Prior work that motivates the gap. Two papers show that sequential inter-slice modelling helps but is expensive to train:

  • ReCoGNet feeds DeepLabV3 slice features into a trained convolutional LSTM along the depth axis of prostate MRI. It reports the memory burden of unrolling hidden states and leaves bidirectional processing to future work.
  • Chen et al. pair a multi-scale 2D network with a trained bidirectional convolutional LSTM to handle anisotropic electron microscopy. They name training complexity as an open problem.

Both learn the recurrence; we freeze it. The underlying failure is documented by Roth et al., who show that independent 2D slice predictions fragment the pancreas across neighbouring slices.

Prior work that supplies the method.

  • Echo state networks keep a fading, unique trace of past input when the recurrent weights are scaled to spectral radius below one (Gallicchio et al.; Gonon et al.).
  • Because reservoir states do not depend on labels, they can be computed once and reused across readouts and validation folds (Lukosevicius et al.).
  • Bidirectional deep echo state networks run forward and backward fixed reservoirs, compress the states with principal component analysis, and train a small MLP. They report training up to 70 times faster than a GRU with close accuracy. That work is restricted to classifying 1D vector sequences; we extend it to dense, spatially structured prediction.
  • Random Controlled Differential Equations show that large random continuous-time systems act as reservoirs that handle irregular sampling. We reuse their continuous-time view so that physical depth, not slice count, sets the dynamics. They tested only 1D time series.

Prior work that constrains the design.

  • Swin-UMamba finds that learned state-space models overfit small medical datasets without ImageNet pretraining. This motivates freezing the dynamics.
  • The Mamba analysis finds that a 1D scan alone is insufficient for 3D context unless augmented with depthwise 3D convolutions. We therefore cannot treat each in-plane pixel as an isolated time series; the reservoir needs some fixed in-plane coupling.
  • The high-dimensional readout overfitting noted in the bidirectional ESN paper is why we compress features before the reservoir and regularise the readout.

Method

Frozen encoder. A frozen 2D encoder produces a feature map for each slice. We use DeepLabV3 on prostate MRI to match ReCoGNet exactly, and SAM's image encoder on multi-organ CT, following the frozen-backbone setting of MA-SAM and ProMISe. Features are reduced to $d$ channels (around 32) with PCA fitted on training slices.

Reservoir update. Let $x_k$ be the reduced feature map of slice $k$, $h_k$ the reservoir state at every in-plane location (around 128 channels each), and $\Delta z_k$ the physical spacing in mm between slices $k-1$ and $k$. The forward reservoir is

$$h_k = (1-\alpha_k)\odot h_{k-1} + \alpha_k \odot \tanh\big(W * h_{k-1} + U x_k\big), \qquad \alpha_k = 1 - e^{-\Delta z_k/\tau},$$

where:

  • $W$ is a fixed random 3×3 depthwise-plus-sparse channel-mixing convolution, rescaled to spectral radius below one so that fading memory holds.
  • $U$ is a fixed random input projection.
  • $\tau$ is a vector of per-channel memory lengths in mm, spread log-uniformly over roughly 1 to 50 mm so that both thin boundaries and whole organs are represented.

The leak $\alpha_k$ comes from exactly integrating a leaky continuous-time unit over a gap of $\Delta z_k$. It is the step that makes the state a function of physical depth rather than slice count. A backward reservoir with independent random weights runs from the inferior end, so both organ boundaries see accumulated context. This addresses the gap ReCoGNet leaves open.

Readout and training. Forward state, backward state and encoder feature are concatenated per slice and fed to a small trained 2D decoder head. A ridge-regression readout at feature resolution is a cheaper variant. Training samples individual slices, and all states are precomputed once and cached, so the training cost equals that of a 2D head. Reservoir hyperparameters (spectral radius, input scale, $\tau$ range) are chosen by grid search against a fast ridge readout. Cached states make this search cheap, which partly answers the hyperparameter-sensitivity problem noted for random reservoirs.

Research Program

The work has three stages.

  1. Accuracy and memory. On small-data prostate MRI and multi-organ CT, we test whether the frozen reservoir closes the gap between plain 2D inference and trained recurrent slice models. We measure peak training memory and wall-clock time alongside accuracy.
  2. Slice-thickness shift. We train at one slice spacing and test at others. Shifts are built by resampling and by dropping slices along depth on each benchmark, so the ground truth stays exact.
  3. Mechanism. We ask which ingredient matters: randomness and nonlinearity, physical-depth stepping, bidirectionality, or in-plane coupling. Each is removed in turn.

Baselines

Naive reference. The do-nothing reference is the same frozen encoder and the same trained 2D head applied slice by slice, with the reservoir states replaced by zeros. On PROMISE12 the corresponding published number is the DeepLabV3 baseline in ReCoGNet: 0.78 Dice on clean data and 0.70 under a 20% contrast drop in the latter half of the volume. We rerun it under our protocol rather than relying on the published figure.

Trained-recurrence competitor. The strongest competitor on the core question of whether training the recurrence is necessary is ReCoGNet itself, a trained unidirectional ConvLSTM over DeepLabV3 features. It reports 0.82 Dice on clean PROMISE12 and 0.78 on the corrupted variant, with 26 training volumes. We also train a bidirectional ConvLSTM in the style of Chen et al., because our reservoir is bidirectional and the comparison should not reward bidirectionality alone. All recurrent baselines use the same frozen encoder, the same 26 training volumes, the same head, and the same hyperparameter-search budget measured in GPU-hours. Beating them means matching their Dice within one standard deviation across five seeds at clearly lower training memory, or exceeding them on the corrupted and thickness-shifted variants.

Multi-organ CT competitors.

  • MA-SAM, a frozen SAM with trained 3D convolution adapters, is the closest parameter-efficient competitor. The file gives only relative figures on BTCV (a 0.9% Dice lead over nnU-Net), so we rerun it ourselves.
  • On AMOS22, SegVol reports 0.8593 average Dice. On TotalSegmentator, the Mamba analysis reports 88.01 average Dice.

These are full-3D, heavily trained upper references. We do not expect to beat them in-distribution, and we report the gap honestly alongside training memory.

Ablations. Four ablations isolate the key ingredients:

  • Index-step reservoir: replace $\Delta z_k$ with a constant, so the reservoir counts slices instead of millimetres.
  • Deterministic filter bank: replace the random nonlinear reservoir with a bank of linear exponential smoothers at the same $\tau$ values in mm.
  • Unidirectional: drop the backward pass.
  • No in-plane coupling: set $W$ to channel-only mixing.

Why not just…?

Why not just stack a few neighbouring slices as input channels (2.5D) and train the head? This is the most serious practical alternative. It also avoids backpropagation through depth and is trivially cheap. Its context, however, is a fixed number of slices, so it sees a few millimetres at 1 mm spacing and a few centimetres at 5 mm spacing. Its behaviour should therefore change under thickness shift, and it cannot see an organ's distant boundary. We run 2.5D stacking with three, five and nine slices as a baseline under identical training. If it matches the reservoir both in distribution and under thickness shift, the reservoir is unjustified and we will report that.

Why not just train a ConvLSTM with gradient checkpointing or truncated backpropagation to fix the memory problem? Checkpointing trades memory for recomputation, and truncation shortens the context the model can learn from. Neither removes the overfitting risk that Swin-UMamba documents for learned sequence dynamics on small cohorts. We include a checkpointed ConvLSTM at matched GPU-hours. The decisive evidence is the comparison at 26 PROMISE12 volumes and at reduced training sets of 13 and 6 volumes, where fixed dynamics should overfit less.

Why not just smooth the 2D predictions along depth afterwards? Post-hoc smoothing of probability maps along depth removes isolated dropouts cheaply. It cannot recover a structure that the 2D model missed on several consecutive slices, and it cannot use feature-level evidence from neighbouring slices. We run Gaussian depth smoothing in mm plus largest-connected-component filtering on the naive baseline's outputs. If it closes most of the Dice and fragmentation gap, the problem was mostly cosmetic and our claim narrows accordingly.

Why not just use fixed exponential smoothers of features instead of a random nonlinear reservoir? This is a real threat, because a linear leaky reservoir is essentially such a filter bank. The random nonlinear mixing is justified only if it creates feature interactions across depth that the readout can exploit, as the kernel view of reservoirs suggests (Calandra et al.). The deterministic-bank ablation settles this directly. If the bank ties the reservoir, the paper's contribution becomes "frozen physical-depth feature integration," which we would state plainly as the simpler and better result.

Proposed Experiments

Datasets. PROMISE12 serves as the low-data testbed with its 26-volume training split, in both clean and 20%-contrast-corrupted variants as defined by ReCoGNet. AMOS22 and TotalSegmentator are the multi-organ CT benchmarks. The ISBI neuronal electron microscopy challenge is the strongly anisotropic case; Chen et al. report a Rand score of 0.9753 there with a trained bidirectional ConvLSTM.

Metrics.

  • Dice and average surface distance.
  • A fragmentation count adapted from the failure Roth et al. describe: the number of times a ground-truth-continuous structure's predicted mask disappears and reappears along depth.
  • Peak training GPU memory and training wall-clock time.

Check 1: low data. On PROMISE12 and its corrupted variant, the reservoir should exceed the naive frozen-2D head and the post-hoc smoothing baseline by a clear margin. It should come within seed variance of ReCoGNet's 0.82 and 0.78 at substantially lower training memory.

Check 2: slice-thickness shift. On AMOS22 and TotalSegmentator, we train with volumes resampled to 1.5 mm spacing and test at 3 and 5 mm. The prediction is that the physical-step reservoir loses less Dice than the index-step ablation, 2.5D stacking, the ConvLSTM, and MA-SAM. If the index-step ablation degrades as little as the physical-step version, the continuous-depth claim is false.

Check 3: mechanism. The ablations decide whether randomness and nonlinearity add anything beyond fixed smoothing, and whether bidirectionality reduces fragmentation specifically at the superior and inferior organ ends.

Reporting. Every result is averaged over five seeds, including five independent reservoir draws, so variance due to the random weights is reported rather than hidden.