Contrast-Normalized Scattering for Single-Source Cross-Modality Segmentation

A segmentation network trained on one medical imaging modality, such as MRI, usually fails badly on another, such as CT. The anatomy is the same, but the mapping from tissue to brightness is different and is often not even monotone: fluid is bright in one MRI sequence and dark in another, and bone is bright in CT yet nearly invisible in MRI. This note proposes a fixed, untrained image representation that removes this mapping by construction at most anatomical boundaries. It is built from the wavelet scattering transform, a cascade of fixed multi-scale oriented filters, absolute values, and local averaging that is known to be stable to small deformations. Its responses are then divided by their own local energy. A standard segmentation network trained on top of it should transfer across modalities without generating synthetic target images.

The idea fits in two sentences. Zero-mean filters ignore constant brightness offsets, and the absolute value ignores which side of an edge is brighter. Dividing by local energy then cancels how strong the edge is, so at any boundary between exactly two tissues the representation is the same whatever brightness each tissue has. The proposal turns this observation into an exact statement, builds the representation, and tests where it holds and where it predictably breaks, at points where three or more tissues meet and in noisy flat regions. This is set against the intensity-augmentation and diffusion-based methods that currently lead single-source cross-modality generalization.

Core Claim

Let $x=\sum_k c_k \mathbf{1}_{\Omega_k}$ be an idealized image made of tissue regions $\Omega_k$ with intensities $c_k$, and let $y=\sum_k g(c_k)\mathbf{1}_{\Omega_k}$ be the same anatomy imaged under an arbitrary, possibly non-monotone, tissue-to-intensity map $g$. Write $\psi_\lambda$ for a zero-mean wavelet at scale and orientation $\lambda$ and $\phi_J$ for an averaging window of width $2^J$. Near a point $u$ whose neighborhood of radius about $2^J$ touches only regions $a$ and $b$, $x\star\psi_\lambda = (c_b-c_a)(\mathbf{1}_{\Omega_b}\star\psi_\lambda)$, because the zero-mean filter removes $c_a$. Likewise $y\star\psi_\lambda=(g(c_b)-g(c_a))(\mathbf{1}_{\Omega_b}\star\psi_\lambda)$. After the absolute value the two images differ only by a positive factor. That factor cancels in the normalized coefficients

$$\tilde S_1 x(u,\lambda_1)=\frac{|x\star\psi_{\lambda_1}|\star\phi_J(u)}{\sum_{\lambda'}|x\star\psi_{\lambda'}|\star\phi_J(u)+\epsilon},\qquad \tilde S_2 x(u,\lambda_1,\lambda_2)=\frac{\big||x\star\psi_{\lambda_1}|\star\psi_{\lambda_2}\big|\star\phi_J(u)}{|x\star\psi_{\lambda_1}|\star\phi_J(u)+\epsilon},$$

where $\epsilon$ is a noise floor estimated per image. So $\tilde S x=\tilde S y$ exactly at two-tissue boundaries for any $g$, including contrast inversion.

The zeroth-order term $x\star\phi_J$ carries the brightness itself, which is the modality-specific part, and it is discarded. The claim is therefore narrower and more testable than "scattering gives modality invariance." The invariance is exact away from junctions of three or more tissues. It degrades in a quantifiable way near those junctions, where the ratios of the jumps change with $g$. It also degrades in textured or noisy regions, where the piecewise-constant model fails. If this approximate invariance covers most organ boundaries, it should deliver most of the cross-modality gain without synthesizing target-style images.

Each part has a role. Deformation stability, inherited from the scattering construction of Multiscale Scattering for Audio Classification and the linearization results of Morgenshtern et al., absorbs the small geometric differences between scanners. Normalization absorbs gain. The modulus and zero-mean filters absorb sign and offset.

Relation to Prior Work

Three lines of work motivate the gap. ScatterFormer adds a scattering branch to a transformer to keep high-frequency EEG spikes under patient-to-patient shift. That is evidence that scattering features generalize across subjects, but it addresses neither imaging modalities nor dense segmentation, and it does not use the sign- and gain-cancelling structure argued for here. cWDM uses a fixed discrete wavelet transform as a training-free latent space for synthesizing missing MRI sequences. Its linear coefficients flip sign and rescale when contrast changes, which is exactly the property this proposal removes. The linear wavelet transform therefore becomes an ablation rather than a competitor. Deep Network Classification by Scattering and Homotopy Dictionary Learning shows that fixed scattering plus a single learned sparse layer beats AlexNet on ImageNet but lags deep networks. This result constrains the design: the scattering stage should be a front end to a full learned U-Net, not a replacement for it.

The methods being competed against sit in cross-modality generalization. Zhou et al. model the modality shift as a random non-linear brightness curve, implemented with Bézier curves, and route test images through separate normalization branches. Chen et al. synthesize target-style images with Stable Diffusion and ControlNet and align features contrastively. They report hallucinated vertebral arches, a failure mode that a fixed representation cannot have. AdaptDiff and DSAN are unsupervised adaptation methods: they see unlabeled target images, so they serve as reference points rather than like-for-like baselines. URMMDG learns modality-invariant features with a mutual-information penalty. We replace that learned separation with an analytic one, but only for the brightness-mapping part of the shift.

Research Program

The first component is analytical. We will make the two-tissue invariance precise for smooth boundaries whose curvature radius exceeds $2^J$. We will bound the error near junctions as a function of the jump ratios. We will also show how additive noise of standard deviation $\sigma$ inflates the normalized coefficients in flat regions, which justifies setting $\epsilon$ proportional to $\sigma$ estimated from the finest wavelet scale.

The second component is the representation itself. It is an undecimated 2D scattering transform, computed slice-wise and later in 3D, with Morlet wavelets, $J\in\{2,3\}$ and 8 orientations. It is computed at full resolution so that segmentation boundaries are not lost. We drop $S_0$, compute $\tilde S_1$ and $\tilde S_2$ with the noise-adaptive $\epsilon$, and use the result as the input channels of a standard U-Net.

The third component is a measurement tool. We will apply random label-wise brightness maps, including non-monotone ones, to labeled source images. For each pixel we then measure the representation distance between the original and remapped image as a function of distance to the nearest three-tissue junction. This gives a direct, model-free test of the claimed invariance class before any segmentation is trained.

Baselines

The naive reference is the DeepAll model: the same U-Net trained on raw source-modality intensities with the same geometric augmentation and no modality-specific treatment. For BraTS with T2 as the single source, Zhou et al. report that their method reaches 54.44% Dice, a 19.27% improvement over DeepAll. We will rerun DeepAll ourselves rather than infer its value from that gap.

The strongest like-for-like baseline on BraTS and cross-modality cardiac data is Bézier augmentation with dual normalization Zhou et al.. It targets exactly the same nuisance, an unknown brightness map, but handles it by sampling instead of cancelling it analytically. Its reported 54.44% Dice on BraTS T2-source single-source generalization is the number to beat. We will reimplement it on our U-Net with the same splits, epochs, three seeds and an equal hyperparameter-search budget. Beating it means a higher mean Dice across the unseen BraTS sequences, by more than the seed standard deviation, under that matched protocol.

On abdominal CT to MRI (AMOS), the strongest existing method is the diffusion-based causal intervention of Chen et al., reported at 86.20% average Dice, within 1.13 points of target supervision. On lung CT to X-ray the same paper reports 78.79% versus a 67.33% baseline. Rerunning its Stable Diffusion pipeline at matched compute is the hard part of fairness. We will therefore report its published numbers on its protocol alongside our own matched-backbone runs. Success here means coming within about two Dice points at a small fraction of training compute, or exceeding it when our representation is combined with its contrastive loss.

For cardiac MRI to CT, DSAN reports 78.50% Dice, but it uses unlabeled CT. It is an upper reference, not a target.

The ablations isolate the key ingredient, normalization. The first is unnormalized scattering with $S_0$ kept. The second is unnormalized scattering with $S_0$ dropped. The third is the linear wavelet transform used in cWDM as the input. The fourth is first-order coefficients only. The fifth is a single-scale normalized gradient magnitude. All are fed to the identical U-Net.

Why not just…?

Why not just use heavier random intensity augmentation? This is the most serious threat. Bézier-style remapping already targets the same nuisance, and with enough random brightness curves a U-Net could learn the invariance itself. The analytical argument says augmentation can only approximate an invariance that the normalized representation has exactly at two-tissue boundaries. Whether exactness matters in practice is empirical. We will train both with augmentation intensity swept upward and report the curves. If heavy augmentation alone matches the normalized representation, we will say so. The proposal's remaining value would then be the junction-level diagnosis of where any brightness-based method fails, plus the combination of the two approaches.

Why not just feed a normalized edge map? A gradient magnitude divided by local contrast is also sign- and gain-invariant, and it is what our first-order coefficients reduce to at one scale. If it performs as well as the full representation, the multi-scale and second-order machinery is unnecessary. The single-scale normalized-gradient ablation exists to settle this. We expect it to lose specifically on textured targets and thin structures, where second-order coefficients separate texture from boundary. We will report that comparison separately by structure type rather than only as average Dice.

Why not just fine-tune a segmentation foundation model? SSM-SAM shows that the Segment Anything Model degrades on medical images and needs adapters, prompt modules and meta-learning to recover. Separately, Wolf et al. find that large pretrained vision-language models lean on memorized anatomical priors. Neither makes a foundation model invariant to contrast inversion. We will nonetheless include a SAM-adapter run trained on source only as an additional comparison on AMOS, because scale could plausibly absorb the shift.

Why not just generate target-style images with diffusion? This is the approach of Chen et al.. It works well on AMOS, but its authors document hallucinated structure, and it needs a text prompt describing the target modality. A fixed representation cannot hallucinate anatomy. We will measure whether that safety comes at a Dice cost by reporting boundary-distance error on spine and lung, where hallucination was observed.

Proposed Experiments

The first decisive check is the synthetic invariance map described in the research program. Normalized scattering distance should be near zero beyond about $2^J$ from three-tissue junctions and should rise near them. Unnormalized scattering and the linear wavelet transform should show large distances everywhere under sign-flipping maps. If normalized distances are not small away from junctions on real MRI, the core claim is false, and we stop before training segmenters.

The second check is single-source generalization. The settings are BraTS with T2 as source and T1, T1ce and FLAIR as targets; MM-WHS cardiac MRI to CT and CT to MRI; AMOS CT to MRI; and Chen et al.'s lung CT to X-ray task. For each we report Dice and average surface distance over three seeds against all baselines above.

The third check tests the mechanism rather than the score. For every target structure we compute the fraction of its boundary lying within $2^J$ of a three-tissue junction. The theory predicts that the per-structure Dice gain over DeepAll falls as this fraction rises. It also predicts that augmentation-based methods show a weaker dependence of this kind. A clear negative correlation would confirm that the gains come from the stated invariance and not from generic filtering. Absence of the correlation would mean the representation helps for some other reason, which we would then report as such.

Finally, we measure robustness to the noise floor by sweeping $\epsilon$ and adding modality-typical noise. The aim is to confirm that the noise-adaptive choice avoids amplifying flat-region noise, which is the obvious way this design could fail on low-dose CT.