A segmentation network trained on MRI and run on CT mostly fails at the edges, not in the middle. The interior of the heart or liver still looks like an organ, but where two tissues have similar brightness in the new scanner type, the network's per-voxel guesses become confident and wrong. The fact that decides those edges does not depend on the scanner. Left-ventricle blood pool is wrapped by myocardium in every modality, and the liver touches the kidney and not the lung. This proposal writes that fact down as a Markov Random Field (MRF), a probabilistic model that scores a labeling by how plausible each pair of neighboring labels is. The MRF is fitted only to source-modality label maps and then frozen. On an unlabeled target modality it serves as the objective for a tiny test-time recalibration of the network's per-class confidences.
The claim has two parts. First, class-adjacency statistics transfer across modalities far better than intensity statistics. Second, they are a safer self-supervision signal than the whole-image class-volume priors used by current test-time adaptation, because pathology changes how much of each tissue there is but rarely which tissues touch. If both hold, the gains should show up mainly in surface-distance error at low-contrast boundaries, and most strongly in the hard transfer direction, CT→MRI.
What exists and what is missing
The mechanism comes from An MRF-UNet Product of Experts for Image Segmentation. That work multiplies a 3D U-Net's output with an MRF prior over label transitions, solved by a recurrent mean-field layer. Because the prior ignores image intensity, it improved out-of-distribution brain MRI Dice from 0.67 to 0.73. It was tested only T1→T1 across scanners, and the prior is used passively at inference. We reuse its layer and ask two new things: whether the same label-only prior survives a change of physics (MRI↔CT, bSSFP→LGE/T2), and whether it can drive adaptation rather than only smooth.
Two lines of cross-modality work motivate the gap. The first is Gradient-Map-Guided Adaptive Domain Generalization for Cross Modality MRI Segmentation. It replaces intensities with Sobel gradient maps and adapts at test time with class-prior reweighting (PITTA). On MS-CMRSeg2019 bSSFP→T2 it lifts Dice from 4.9% to 85.4%. Its own limitation is that the class priors are volumetric ratios, which break for patients with large pathology or resections. The second is Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation, which aligns landmarks and Canny edges on MM-WHS. It reports a sharp asymmetry: 85.8% Dice MRI→CT versus 74.8% CT→MRI.
Both methods rely on image-derived edges, and image-derived edges vanish exactly where contrast is low. The same weakness constrains classical contrast-sensitive MRFs: lowering the pairwise penalty at strong gradients does nothing when there is no gradient. Our pairwise term therefore never looks at the image.
The generative route is Cross-Modality Knowledge Transfer for Prostate Segmentation from CT Scans, which uses an SSIM-constrained CycleGAN (0.45→0.73 Dice). AdaptDiff uses conditional diffusion on pseudo-labels (73.16%→81.94% on OCTA-500). Both papers document that image translation distorts or drops fine boundaries. We treat translation as a competitor rather than a component. The CHAOS analysis (CHAOS Challenge) sets the evaluation lens: cross-modality Dice drops from about 0.97 to 0.88, and maximum surface distance stays at 20–30 mm even when Dice exceeds 0.95. Dice alone would hide the effect we predict.
Core mechanism
Let $z_i \in \mathbb{R}^K$ be the source-trained network's logits at voxel $i$ over $K$ classes. Let $W_d \in \mathbb{R}^{K\times K}$ be the log-frequency of label pair $(k,l)$ at neighbor offset $d$, estimated from source ground-truth maps. The offsets are direction-specific, so that "myocardium lies outside blood pool" can be encoded. The target posterior is the mean-field solution $q$ of
$$E(y) = -\sum_i \big(a \odot z_i + b\big)_{y_i} - \lambda \sum_{i}\sum_{d} W_d[y_i, y_{i+d}],$$
where $a, b \in \mathbb{R}^K$ are per-class scale and bias applied to the logits and $\lambda$ weights the prior. The prior terms $W_d$ and $\lambda$ are fixed from the source. Only $a$ and $b$, that is $2K$ numbers per target modality or volume, are fitted at test time. They are chosen by minimizing the mean-field free energy restricted to a band of voxels near the current predicted boundary.
Collapse is a real risk: an all-background labeling is highly probable under adjacency statistics too. We guard against it with two constraints. Scales $a$ are constrained positive. Voxels whose source-model entropy is in the lowest decile are held as fixed anchors. This replaces PITTA's volume prior with a much weaker assumption, namely that confident interiors are mostly right. Whether that assumption holds under large shift is one of the things the experiments test.
Research program
The first step is a cheap falsification check that needs no training. On MM-WHS and CHAOS, we compute the directional adjacency matrices $W_d$ from MRI ground truth and from CT ground truth separately. We then measure their divergence against the divergence of per-class intensity histograms. Target labels are used here only for analysis. If adjacency statistics differ between modalities about as much as intensities do, the premise fails and we stop.
Second, we train source models under three input choices: raw intensities, the gradient-map input of the gradient-map paper, and the MRF-UNet product. This separates "better input features" from "better prior."
Third, we apply the frozen-prior recalibration to each source model on the target modality.
Fourth, we build a contrast-stratified boundary evaluation. For every ground-truth boundary voxel in the target, we measure the local intensity difference across the true boundary. We bin voxels into contrast quintiles and report boundary error and surface distance per bin. The mechanism predicts gains concentrated in the lowest bins. It also predicts larger gains CT→MRI than MRI→CT, since MRI has the less calibrated contrast.
Baselines
The naive reference is the source-only network applied to the target modality. The file reports it as 4.9% Dice on MS-CMRSeg bSSFP→T2 (Gradient-Map-Guided) and, without domain adaptation, as 64.9% (MRI→CT) and 38.1% (CT→MRI) on MM-WHS (Structure-Driven UDA). We re-run it with our backbone rather than copy these numbers.
The strongest test-time competitor is gradient-map input with PITTA, at 85.4% Dice on bSSFP→T2. We re-implement it on the same backbone, the same source data and an equal test-time compute budget, and tune its hyperparameters on a held-out source split only, never on target labels. The CHAOS "peeking" analysis shows that target-side tuning inflates scores. Beating it means matching its Dice on MS-CMRSeg while lowering HD95 in the bottom two contrast bins. On BraTS sequence transfer, where tumor volume varies widely, the proposal should beat it outright.
The strongest unsupervised-adaptation competitor is the structure-driven landmark method on MM-WHS, at 85.8% (MRI→CT) and 74.8% (CT→MRI) Dice. It uses target images during training, whereas we only adapt at test time. Our success criterion is therefore narrowing the CT→MRI gap, not exceeding 85.8% MRI→CT. We will report whichever outcome occurs.
The passive-prior reference is MRF-UNet without any recalibration. Its reported cross-scanner result is 0.73 versus 0.67 Dice, but no reported result was found on Lacuna for it across modalities. For generative transfer we run an SSIM-loss CycleGAN pipeline following the prostate paper, which reports 0.73 Dice on real CT.
Three ablations isolate the key ingredient. The first recalibrates $(a,b)$ by plain entropy minimization with no MRF. The second uses the MRF with image-contrast-sensitive pairwise weights instead of label-only ones. The third uses isotropic rather than directional $W_d$.
Why not just…?
Why not just add a contrast-sensitive conditional random field as post-processing? Such a field lowers the smoothing penalty wherever the image has a strong edge. That helps when the boundary is visible and does nothing when it is not. The limitation noted for MRF smoothing in the file is exactly this failure at low gradient magnitude. The second ablation tests it directly. If the contrast-sensitive version matches ours in the lowest contrast bins, the label-only claim is wrong.
Why not just use the gradient-map input and plain test-time entropy minimization? This is the real threat. Gradient maps already deliver most of the invariance on MS-CMRSeg, and entropy minimization is simpler than our objective. The first ablation runs exactly this combination. Our bet is that entropy minimization without a structural prior drifts toward dominant classes, which is the collapse PITTA's class priors were added to prevent. We further bet that the adjacency prior prevents it without assuming organ volumes. BraTS, with its variable tumor burden, is where the difference should appear. If it does not, the contribution shrinks to "a cheaper replacement for class priors" and we will say so.
Why not just translate images with a better generator? Translation methods need training on target images and pay a boundary-distortion cost that both AdaptDiff and the prostate CycleGAN paper report. Our method needs only $2K$ numbers per target and no target training. The CycleGAN baseline tells us whether that economy costs accuracy.
Why not just keep the largest connected component or apply morphological cleanup? These operations remove specks but cannot move a boundary that sits in the wrong place. They also cannot express which classes may touch, for example blood pool meeting background. We include connected-component cleanup on top of the source-only model to show that it changes Dice slightly and leaves maximum surface distance unchanged.
Decisive checks
Evaluation uses MM-WHS in both directions, MS-CMRSeg bSSFP→LGE and bSSFP→T2, CHAOS CT↔MR, and BraTS single-sequence transfer. We report Dice, HD95, average and maximum symmetric surface distance, and the per-contrast-bin boundary error.
The proposal succeeds if three things all hold. Adjacency statistics diverge across modalities much less than intensity statistics. Frozen-prior recalibration lowers surface distance mainly in the lowest-contrast bins, with a larger effect CT→MRI. On BraTS it stays stable where volume-prior adaptation degrades.
It fails if the gains are uniform across contrast bins, because that would mean generic smoothing rather than anatomy is doing the work. It also fails if entropy minimization on gradient-map inputs matches it everywhere.
References
- An MRF-UNet Product of Experts for Image Segmentation
- Gradient-Map-Guided Adaptive Domain Generalization for Cross Modality MRI Segmentation
- Structure-Driven Unsupervised Domain Adaptation for Cross-Modality Cardiac Segmentation
- Cross-Modality Knowledge Transfer for Prostate Segmentation from CT Scans
- AdaptDiff: Cross-Modality Domain Adaptation via Weak Conditional Semantic Diffusion for Retinal Vessel Segmentation
- CHAOS Challenge - combined (CT-MR) healthy abdominal organ segmentation