Anatomy-Driven Token Trees for CT–MRI Transfer

Transformers for medical images cut a scan into small square or cubic patches and treat each patch as a token. Attention cost grows with the square of the token count, so fine patches are expensive. Recent work avoids this cost by spending fine patches only where the image is detailed and leaving coarse patches over flat background. The usual rule for "detailed" is image edges, and that rule fails when a model trained on one imaging modality is used on another. A CT scan and an MRI scan of the same heart have very different edge maps: bone and contrast-filled vessels are sharp in CT, while MRI shows soft, noisy intensity gradients. An edge-driven patching rule therefore gives the two modalities different token layouts before the network has seen anything. Under modality shift, the choice of where to spend fine tokens is itself a source of domain gap.

We propose to make that choice from predicted anatomy instead of raw edges. A hierarchical transformer first processes the scan with coarse patches and predicts where organ boundaries probably lie. Only tokens near those predicted boundaries are split into finer patches for later stages, under a fixed token budget. The bet is that coarse, context-rich predictions transfer across modalities better than local edge statistics do. If so, deriving the token layout from them should give the same layout for CT and MRI of similar anatomy, and should put fine compute where boundary errors happen. A small composite loss keeps the layout stable across appearance changes and keeps the fine tokens focused on structure rather than modality-specific intensity.

Core Claim

In unsupervised CT↔MRI segmentation transfer, a token tree split by the network's own coarse boundary prediction will beat an edge-split tree at equal token budget. Here "unsupervised" means labels exist only in the source modality. The gain should appear in the target modality, it should be largest on boundary-distance metrics, and it should be explained by the tree placing more of its fine tokens near true boundaries in the target modality.

The closest existing work is SHF (SHF). SHF builds quadtree and octree patchings from blurred Canny edges, orders the variable-size patches along a Z-order curve, and decodes masks by reversing the tree. It reports large speed and Dice gains on pathology, BTCV and KiTS, but only within a single modality. We reuse its tree tokenization and depatching. We replace the fixed edge heuristic with a learned, anatomy-conditioned split and test it under modality shift, which SHF never does. The idea also differs from the transformer-native cross-modality model C-ViT (C-ViT). C-ViT swaps LayerNorm for modality-conditioned instance normalization in a Swin-UNETR, uses a fixed window grid, and trains with labels from both modalities. It differs again from CARVE (CARVE), which compresses tokens at inference without training, for 3D language models rather than dense segmentation.

Research Program

The backbone has three stages. Stage one embeds the volume in large patches, for example 16³ voxels, and runs full attention over this short sequence. A light head on stage one outputs a boundary probability per token. We use a dedicated boundary head rather than prediction entropy because the BRAVO challenge results (BRAVO) show that confidence calibration degrades with finer tokenization, and calibration is known to suffer further under shift. Using entropy is kept as an ablation. The top-ranked tokens are split into eight children, up to a budget $K$; stage two repeats the process, and stage three operates on the resulting leaves. Splitting is a hard top-$K$ selection with no gradient through the choice. The boundary head is trained directly with a boundary target from source labels, so it needs none. Decoding follows SHF's reverse-tree depatching.

Children inherit their parent's stage-one features through a residual connection. This addresses the concern raised by Luka et al. that small tokens carry too little context to be useful early on. Because a child only exists inside a parent that was already processed coarsely, it never has to rebuild context from scratch.

The training objective has three terms:

$$ \mathcal{L} = \mathcal{L}_{seg}(x_s, y_s) + \lambda_c \sum_{\ell=1}^{3} \mathcal{L}^{(\ell)}_{cons}\big(x, \tilde{x}\big) + \lambda_r \mathcal{L}_{rec}(x). $$

The first term, $\mathcal{L}_{seg}$, is Dice plus cross-entropy on labeled source scans $x_s$ with labels $y_s$, applied at every stage, in the spirit of MERIT's multi-stage supervision (MERIT).

The second term, $\mathcal{L}^{(\ell)}_{cons}$, is a consistency loss at stage $\ell$. It compares the predictions and boundary scores of a scan $x$ with those of an appearance-altered copy $\tilde{x}$. Two kinds of alteration are used: random nonlinear intensity remappings, including contrast inversions that mimic CT-versus-MRI tissue ordering, and, in one variant, an offline CycleGAN translation. The loss is applied to source scans and to unlabeled target scans. Because the boundary score drives the split, this term directly pushes the tree layout to stay the same when appearance changes.

The third term, $\mathcal{L}_{rec}$, asks each leaf token to reconstruct a local self-similarity descriptor of its region, a structure code that compares patches within the same image. Reconstructing raw intensity is the obvious alternative, but it would push fine tokens to encode exactly the modality-specific appearance we want to discard. This mirrors the lesson from SINCO (SINCO), which added a segmentation-driven structural term to a pixel reconstruction loss so that compression kept diagnostic boundaries.

The weights $\lambda_c$ and $\lambda_r$ are tuned on a source-only validation split. This avoids leaking target labels and limits the loss-balancing instability that composite objectives are known for. The analysis also asks whether stage-one coarse predictions really transfer better than fine-stage ones. If they do not, the premise fails, and we will report that.

Baselines

The naive reference is a model trained on source labels and tested on the target with no adaptation. SIFA (SIFA) reports 17.2% Dice for this setting on MM-WHS MRI→CT. We rerun it with our own backbone using a uniform coarse patch grid, so the floor is measured on the same architecture.

The strongest published methods are CNN adaptation systems, and we compare against their reported numbers on the standard splits. DSAN (DSAN) uses a shared encoder, private decoders for each modality, and bidirectional translation. It reports 78.50% Dice for MRI→CT and 66.45% for CT→MRI on MM-WHS, and it cites SIFA at 74.10%, while SIFA's own paper reports 73.0%; we will quote both. On abdominal CHAOS data, CISFA (CISFA), which uses one-sided translation with organ-weighted patch contrastive losses, reports 76.24% Dice for CT→MRI and 70.45% for MRI→CT. Beating the field means exceeding DSAN on MM-WHS and CISFA on the abdomen in mean Dice, or matching them while clearly improving average surface distance at lower compute.

Because the CNN systems differ from ours in architecture, the comparison that isolates our contribution is internal. We will train a transformer analogue in which DSAN-style translation feeds a uniform-grid Swin-UNETR. This uses the same data, the same augmentation, the same GPU-hour budget, and the same number of hyperparameter trials as our model.

The key ablations hold the token budget $K$ fixed and change only the split rule. The rules are the SHF Canny-edge tree, the Canny tree with per-modality normalized thresholds, a random tree, an entropy-driven tree, and our boundary-head tree. A second set of ablations removes the consistency term, the reconstruction term, or both, and replaces the structural reconstruction target with raw intensity. A win means our tree beats the best edge-based tree in target Dice and surface distance at matched budget, over at least three seeds.

Why not just…?

Why not just tune the Canny thresholds per modality, or normalize intensities first? This is the most serious threat, because it is cheap and could close most of the gap. We include it as a baseline, with thresholds tuned per modality on unlabeled target volumes so that the fraction of split tokens matches. The experiment that settles it is the boundary-recall diagnostic described below. Edge tuning can match how many tokens are split, but not whether they land on anatomical boundaries rather than on MRI noise or field inhomogeneity. If normalized Canny trees reach similar boundary recall and Dice, we will report that the learned splitter is unnecessary and shift the contribution to the loss terms.

Why not just use uniform small patches in Swin-UNETR or C-ViT? Uniform fine patching multiplies attention cost sharply. SHF reports a 32.3× speedup over uniform SAM patching on PAIP and 6× faster training on 3D data, and fine patching causes out-of-memory failures in other restoration work (Structure-Guided Diffusion Transformer). We run a uniform-grid Swin-UNETR at the finest patch size that fits the memory budget and report Dice against FLOPs, so the question becomes empirical rather than rhetorical.

Why not just translate MRI to CT offline with CycleGAN and train any segmenter? SIFA reports CycleGAN-only adaptation at 57.6% Dice on MM-WHS MRI→CT. Zhang et al. (Cycle- and Shape-Consistency GAN) document hallucinated or erased anatomy and instability when offline-translated data is used. We still run CycleGAN translation in front of our tree model to test whether the anatomy-driven tree adds value on top of translation.

Why not just use nnU-Net with heavy augmentation? CHAOS (CHAOS) shows such CNNs excel within one modality but still have maximum surface distances around 21 mm, and they fall from about 0.97 to 0.88 Dice in joint CT-MR settings. We train nnU-Net on source data with the same intensity-remapping augmentation as an extra baseline.

Proposed Experiments

The primary benchmarks are MM-WHS cardiac CT↔MRI and the CHAOS abdominal organs, both in the unsupervised transfer direction. The metrics are Dice, average surface distance, and Hausdorff distance, together with FLOPs and peak memory. A secondary check uses BTCV within a single modality, to confirm that the learned tree does not lose to SHF where edges are reliable; SHF reports a 6.93% Dice gain there over its baseline. A tertiary check repeats the transfer setup in the few-label regime used by SSM-SAM (SSM-SAM) on ABD-30 and ABD-MRI.

The decisive diagnostic is target-modality boundary recall: the fraction of fine leaf tokens lying within 2 mm of a ground-truth boundary in held-out target volumes. The second diagnostic is tree agreement between a source scan and its translated copy. The core claim holds only if three things happen together. Our tree must have higher target boundary recall than every edge-based tree. That recall must correlate with the per-structure surface-distance gains. And the advantage must be largest for thin structures such as the myocardium and pulmonary artery, where SIFA notes its own failures. If the boundary-head tree wins on Dice but not on boundary recall, the gain comes from somewhere other than token placement, and we will say so.