Linear-time sequence models such as Mamba, a state-space model that reads tokens one at a time and carries a compressed memory forward, can only process a 3D brain scan after it has been flattened into a 1D sequence. Every existing flattening rule is blind to what is in the image. Row-by-row sweeps, snake sweeps, fractal space-filling curves and three orthogonal sweeps all visit voxels in an order fixed by the coordinate grid. White matter is not organized along that grid. It is made of curved bundles of axons, and diffusion MRI measures their local direction in every voxel. We propose to use that measured direction as the reading order. The volume is serialized by tracing short paths along the dominant fiber orientations, so that consecutive tokens in the sequence belong, with high probability, to the same anatomical bundle. A state-space model then runs along these fiber-ordered sequences, and its outputs are scattered back to the voxel grid.
In two sentences: grid-based scans make a sequence model's memory jump across bundle boundaries at every step, and that is the wrong bias for segmenting thin, curved white-matter tracts. If the scan instead follows the fibers that the scanner itself measured, the model's memory stays inside one tract. It should then segment tracts like the fornix better than grid scans, 2D slice networks or 3D convolutions at matched compute. The test bed is direct white-matter tract segmentation, where the best-known method still works on 2D slices and is known to struggle with exactly these thin, curved structures.
Background and the gap
Diffusion tensor imaging (DTI) fits a 3×3 matrix to each voxel. The matrix describes how freely water moves in each direction, and its main axis points along the local fiber bundle. Richer models give up to three fiber-orientation "peaks" per voxel so that crossing bundles can be represented. TractSeg segments 72 tracts directly from a 9-channel volume of these peaks. It runs 2D U-Nets on axial, coronal and sagittal slices and then fuses the three views. It beat TRACULA and RecoBundles by 9 Dice points on Human Connectome Project (HCP) data and by 14 on lower-quality clinical data, in about 2 minutes per subject. However, it has trouble with thin, highly curved tracts such as the fornix, because slicing disconnects them. That weakness motivates this proposal.
On the serialization side, three lines of work improve grid scans. Snake scans in ZigMa and PlainMamba remove row-end jumps. Hilbert curves in FractalMamba keep 2D neighborhoods contiguous and add a shift to repair cuts between quadrants. The three axis-aligned sweeps (Tri-scan) in the 3D medical Mamba analysis reach an 87.93 average Dice across AMOS, TotalSegmentator and BraTS. All three supply method we reuse, and all three are content-blind.
The one content-adaptive scheme is TSP-Mamba. It orders pixels by solving a travelling-salesman problem over feature similarity inside 4×4 patches. It is the closest idea, but it differs from ours in three ways. It uses learned feature similarity rather than a physical orientation field. It needs combinatorial path solving that does not scale to 3D volumes. And it only reorders within tiny windows. Our ordering costs about as much as deterministic tractography, is set by the acquisition physics, and spans whole tracts.
The obvious intersection idea is to replace TractoTransformer's Transformer decoder with Mamba to speed up streamline tracking. We deliberately do not pursue it. Each streamline is only a few hundred steps long, and with cached attention the per-step cost is already roughly linear. The 43 minutes on four V100 GPUs is therefore probably dominated by the number of streamlines and the per-step 3D convolution, not by attention. The recurrent tracker of DeepTract also already exists. A "Mamba tracker" risks being an engineering swap with an unclear payoff.
Our proposal inverts the roles in that work. TractoTransformer uses a sequence model to produce paths. We use cheap, imperfect paths to organize a sequence model's computation over the whole volume, and let the network learn to correct for path errors.
Core Claim
For dense prediction on diffusion data, the order in which a state-space model visits voxels should come from the tissue's measured orientation, not from the coordinate grid. Concretely, we compare two models at matched token budget and architecture. One replaces Tri-scan's three axis-aligned sweeps with three orientation-specific sweeps, one per fiber peak. We predict it will improve Dice on thin, curved tracts more than on large straight ones. We also predict the gain will vanish when the tokens inside each fiber path are shuffled. That shuffled control is the decisive test, because it separates "ordering along fibers" from "grouping voxels that belong to the same tract."
Research Program
The serializer takes the 9-channel peak volume. It seeds a fixed grid of white-matter voxels and traces deterministic paths of bounded length (for example 64 steps) along peak $k$, where $k$ is 1, 2 or 3. Each path is split at points where the curvature or the anisotropy threshold is crossed. The paths for peak $k$ are concatenated into one long sequence. Consecutive paths are ordered by the Hilbert index of their seed points, so that the jumps between paths are also local. Voxels that no path visits, mostly gray matter and fluid, are appended in Hilbert order. The result is exactly three sequences per volume, and together they are capped at the same total token count, $3N$ for $N$ voxels, that Tri-scan uses.
The model is the U-shaped Mamba of the 3D medical Mamba analysis, with its 3D depthwise convolutions kept. That paper showed they are essential, and we do not want the result to hinge on removing them. Only the scan order is changed. Each state-space block reads the features along each orientation sequence. Voxels visited more than once have their outputs averaged when scattered back to the grid. At coarser levels of the U-shaped network, the paths are downsampled by pooling along the path. Paths are computed once per subject in preprocessing on the CPU, and we will report that cost separately.
We will also measure a simple diagnostic: the fraction of sequence-adjacent token pairs that lie in the same ground-truth tract. The hypothesis predicts that gains per tract track this fraction. If they do not, the mechanism is wrong even if Dice improves.
Baselines
The naive reference point is the same U-shaped Mamba with a single raster scan. It shows how much any locality-aware ordering matters on this data.
The strongest existing sequence-model competitor is UlikeMamba_3dMT: Tri-scan with 3D depthwise convolutions and multi-scale blocks, from the 3D medical Mamba analysis. That paper reports 90.60 Dice on BraTS 2021 and an 87.93 average across its three benchmarks. No reported result was found on Lacuna for it on white-matter tract segmentation, so we will train it ourselves. It will use identical peak inputs, identical splits, the same number of optimizer steps, and the same hyperparameter search budget as our model (a fixed number of trials over learning rate and depth). A 3D Hilbert scan with a one-voxel shift, following FractalMamba, is a second content-blind competitor under the same protocol.
The strongest domain method is TractSeg, run from its public release on the same subjects. No absolute per-tract Dice for TractSeg was found on Lacuna; the file reports only its 9- and 14-point margins over older methods. A 3D nnU-Net-style convolutional network, sized to match our FLOPs, covers the "just use 3D convolutions" position.
The key ablation is the shuffled-fiber control. It keeps exactly the same set of voxels in each path but randomizes their order within each path. We also run an orientation-as-input control: the Tri-scan model with the peak channels already present, plus an explicit per-voxel encoding of peak direction.
Beating Tri-scan means a mean Dice gain over the 72 tracts that holds across three training seeds. It must be accompanied by a larger gain on a pre-registered set of thin or curved tracts, the fornix and the anterior commissure among them, at no more than 10% extra wall-clock time including path tracing. Losing to the shuffled control would falsify the ordering claim.
Why not just…?
Why not just use Tri-scan or a 3D Hilbert curve? These are the real threat. With 3D depthwise convolutions supplying local context, scan order may matter little, and the 3D Mamba analysis found those convolutions essential. The matched-budget comparison settles this directly. The per-tract analysis predicts where fiber ordering should win: tracts whose orientation is oblique to every axis, and tracts that are thin. If the gains are uniform or absent, we conclude that grid scans suffice for this task.
Why not just a 3D CNN or TractSeg? TractSeg's documented weakness on the fornix comes from slice decomposition, and a 3D CNN removes that decomposition. A large-kernel 3D CNN may therefore close most of the gap. We include one at matched FLOPs. Fiber ordering must beat it specifically on long, thin tracts, where a fixed receptive field has to span many voxels along a curve and a path-following memory does not.
Why not just feed fiber directions as extra input features? The network already receives the peaks as input, so any benefit could come from the information itself rather than from how computation is ordered. The orientation-as-input control and the shuffled-fiber control test this from two sides. If shuffling within paths costs nothing, then what helps is grouping voxels by tract, and a simpler clustering-based token grouping would be the honest conclusion.
Why not just run tractography and a bundle classifier? Classic pipelines (RecoBundles-style) commit to the traced streamlines. TractSeg already showed that direct segmentation beats them, especially on clinical data. Our paths only schedule computation, so a wrong path costs locality, not correctness. The low-quality experiment below tests whether that robustness holds.
Proposed Experiments
The main experiment uses the 105 HCP subjects and the 72-tract reference used by TractSeg, with TractSeg's splits. We report per-tract Dice, the mean over all tracts, and the mean over the pre-registered thin/curved subset, across three seeds.
To stress the dependence on path quality, we degrade HCP to clinical-like data: 2 mm isotropic voxels and a reduced set of gradient directions. We recompute peaks, and hence paths, from the degraded data. This is where TractSeg's advantage over older methods was largest, and also where our paths are noisiest. We then add an "oracle path" variant, in which paths come from the full-quality data but features come from the degraded data. The gap between oracle and degraded paths measures how much of the method's fate depends on path quality.
For multi-site generalization, we train on HCP and test on bundle masks derived from TractoInferno's multi-site curated bundles, restricted to the tracts shared by both label sets. We report Dice and the drop from in-distribution performance, since cross-site shift is a known failure mode, as Wang et al. observed for connectome classifiers.
The decisive checks are three. First, fiber-ordered scanning beats Tri-scan at equal tokens. Second, the shuffled-fiber control loses most of that gain. Third, per-tract gains correlate with the within-tract adjacency diagnostic. If all three hold, scan order should be treated as a physically informed design choice for diffusion data. If the first holds but the second fails, the paper reports that tract-level token grouping, not ordering, is what matters.
References
- TractSeg - Fast and accurate white matter tract segmentation
- ZigMa: A DiT-style Zigzag Mamba Diffusion Model
- PlainMamba: Improving Non-Hierarchical Mamba in Visual Recognition
- Boosting Vision State Space Model with Fractal Scanning
- A Comprehensive Analysis of Mamba for 3D Volumetric Medical Image Segmentation
- TSP-Mamba: The Travelling Salesman Problem Meets Mamba for Image Super-resolution and Beyond
- TractoTransformer: Diffusion MRI Streamline Tractography using CNN and Transformer Networks
- DeepTract: A Probabilistic Deep Learning Framework for White Matter Fiber Tractography
- Diagnosis and prognosis of Alzheimer's disease using brain morphometry and white matter connectomes