Swelling-Free Structure Tensors as a Modality-Invariant Input for 3D Cardiac Segmentation

A segmentation network trained on cardiac MRI and applied unchanged to CT almost always fails. On the standard whole-heart benchmark, accuracy falls to 17.2% average Dice, a measure of overlap between predicted and true regions (SIFA). The intensities differ between the two scanners, and sometimes the contrast is reversed: a tissue that is bright in one modality is dark in the other. The shapes of the boundaries do not change, because the anatomy is the same. A classical image descriptor already captures boundary shape while discarding contrast. This is the structure tensor: at every voxel, a 3×3 positive-definite matrix that records which directions the local edges point in. We propose to use this tensor field, made scale-free and smoothed correctly, as the main input channel of a 3D segmentation network, so that the network sees nearly the same signal whether the scan is CT or MRI.

The word "correctly" is where matrix geometry enters. Building a structure tensor field means averaging tensors over neighborhoods, and resampling thick-slice clinical volumes means interpolating tensors through depth. Both operations are usually done with ordinary arithmetic. Arithmetic averaging of positive-definite matrices causes the "swelling effect": the average of two equal-volume tensors with different orientations has a larger determinant than either of them (Han et al.). At thin boundaries such as the myocardial wall or the great vessels, swelling blurs the crisp orientation signal into an isotropic blob. These are the structures that cross-modality methods and the whole-heart challenge consistently find hardest (MM-WHS challenge). Our claim is that averaging with a swelling-free mean preserves orientation where the two modalities agree and removes contrast where they disagree. If the claim holds, it should narrow the MRI-to-CT gap without any target-modality images.

Core Claim

Let $I$ be a volume, $\nabla I$ its intensity gradient, and $f$ any smooth intensity remapping relating one modality's appearance to another's. The pointwise outer product transforms as

$$\nabla f(I)\,\nabla f(I)^\top = f'(I)^2\,\nabla I\,\nabla I^\top ,$$

where $f'(I)$ is the slope of the remapping at that voxel. The sign of $f'$ disappears, so contrast reversal has no effect. A local positive scale factor remains, and we remove it by dividing each tensor by the cube root of its determinant, which leaves only orientation and anisotropy.

The classical structure tensor then smooths these normalized tensors with a Gaussian, which is a weighted arithmetic mean. By the swelling inequality, the arithmetic mean of determinant-one tensors with different orientations has determinant greater than one. Euclidean smoothing therefore re-inflates the tensors and lets orientation mixing at boundaries look like isotropy. Replacing that mean with a swelling-free weighted Fréchet mean (the geometric average on the curved space of positive-definite matrices) keeps the normalized field on the determinant-one set. In principle this makes the field invariant both to contrast sign and to local contrast magnitude.

We use the Power-Cholesky metric of Chen et al.. It gives closed-form weighted means without eigendecompositions. More importantly here, it stays numerically stable when diagonal entries approach $10^{-30}$. Log-Cholesky failed 100% of the time once entries reached $10^{-3}$, and near-degenerate tensors are exactly what appear in homogeneous regions like the blood pool.

The novelty is not a new metric. Chen et al. evaluated their metrics only on isolated tensor interpolation and low-dimensional covariance classification. The new contribution is the argument, and its test, that a swelling-free mean is what makes a contrast-normalized tensor field modality-invariant at boundaries, together with the use of that field for cross-modality 3D segmentation.

Research Program

The pipeline has three parts.

Tensor field construction. We compute per-voxel gradient outer products and apply a small Euclidean regularization: a tiny-scale integration plus a small multiple of the identity, which makes each tensor strictly positive-definite. We then normalize each tensor by its determinant. Finally, we build a three-scale tensor pyramid with weighted Power-Cholesky Fréchet means instead of Gaussian convolution.

Thick-slice resampling. For volumes with thick slices, the resampling to an isotropic grid is done on the tensor field with the same swelling-free interpolation, rather than resampling intensities first. This is where volumetric context enters. Clinical through-plane spacing is often several times coarser than in-plane spacing (CARVE). Linear interpolation across a wall that is tilted relative to the slice plane is the textbook case for swelling.

Segmentation network. The tensor features feed a native 3D U-Net. We encode each tensor as its upper triangle after the matrix logarithm, which is a flat encoding of an already-correct tensor. Optionally, intensity is added as an extra channel with dropout. We keep a full 3D network, rather than slice-wise models with post-hoc smoothing, because slice-wise predictions fragment organ boundaries (Roth et al.). We also know 2D foundation models do not reason reliably about 3D spatial relations in medical scans (Wolf et al.).

Analysis track. Alongside segmentation, we measure invariance directly. MM-WHS volumes are unpaired, so we cannot register CT to MRI. Instead, we compute a modality-free reference tensor field from each ground-truth label map, using the gradient of its signed distance function. For each modality and each construction (Euclidean versus swelling-free), we then measure the angular error between the image-derived and label-derived principal directions within a few voxels of each boundary. The quantity of interest is the CT-versus-MRI difference in this error. The mechanism predicts that swelling-free smoothing shrinks that difference most at thin structures.

Role of each direction. Swelling is the core mechanism. Volumetric context is the regime where it bites, through 3D neighborhoods and thick-slice interpolation. Cross-modality generalization is the evaluation that decides whether any of this matters.

Baselines

Naive reference points. The do-nothing reference is a 3D U-Net trained on MRI intensities and tested on CT. The cited number for source-only training in this setting is 17.2% average Dice (SIFA). We will reproduce it with our own backbone, because that figure comes from SIFA's slice-based network. We also run the within-modality supervised upper bound. The challenge reports 0.91±0.09 best Dice on CT and 0.87±0.04 on MRI (MM-WHS challenge).

Strongest unsupervised adaptation method. This is SIFA, which combines adversarial image translation with shared-encoder feature alignment and uses unlabeled target images. It reports 73.0% average Dice for MRI→CT on MM-WHS. SIFA uses target data and our base method does not, so we compare in two ways. First, our target-free model against SIFA's cited number. Second, SIFA retrained with our tensor channels against SIFA retrained with intensity only, using identical splits, backbone, and adversarial schedule. Beating SIFA means matching or exceeding 73.0% without target images, or adding a statistically reliable gain over five seeds when tensors are plugged into SIFA. We expect most of any gain to come from myocardium and aorta.

Few-shot method. For few-shot comparison we use Meta-hallucinator, which reports a 6.7-point Dice improvement and 0.9 mm lower average surface distance over MT-UDA at 4 shots. No absolute Dice is given in our file, so the comparison runs on our reproduction at 1 and 4 shots.

Ablation of the key ingredient. The ablation that isolates our contribution is the same pipeline with classical Euclidean Gaussian smoothing and linear resampling of the normalized tensors. Only the averaging rule changes. Two further arms swap in Log-Euclidean means and affine-invariant means. These test whether the benefit belongs to swelling-free averaging in general or to the Power-Cholesky metric in particular. A Log-Cholesky arm checks whether the stability failures reported by Chen et al. actually occur in homogeneous cardiac regions.

Fairness. Every arm uses the same 3D backbone, the same augmentation, the same epochs, and the same hyperparameter search budget, tuned on a held-out source-modality validation split only. Target labels are never used for model selection.

Why not just…?

Why not just use the ordinary Euclidean structure tensor? This is the most serious threat. Euclidean smoothing already removes contrast sign. Its swelling might even help, as a form of "coherence" smoothing that a deep network can learn to undo. The Euclidean-versus-swelling-free ablation settles the question, on both Dice and the boundary angular-error gap. If the two are indistinguishable, the geometric part of this proposal is unnecessary, and we will report the result as a negative finding about swelling in this setting.

Why not just train with aggressive intensity augmentation, including random contrast inversion and random monotone remappings? This also targets contrast invariance and needs no geometry. We found no reported result on Lacuna for this baseline on MM-WHS, so we run it with the same budget. It is not mutually exclusive with our method, so we also test the combination. If augmentation alone matches tensor input, the invariance claim reduces to a data-augmentation claim. The mechanism we propose would then still predict an advantage specifically on thin, through-plane-tilted structures after thick-slice resampling, which augmentation cannot repair.

Why not just use Log-Euclidean means, given that 3×3 eigendecompositions are cheap? For 3×3 tensors, the 35x speed advantage reported by Chen et al. at 512×512 is largely irrelevant, and we will not rest the case on speed. The real question is near-singular tensors. In homogeneous regions, logarithms of tiny eigenvalues become very negative and can dominate weighted means. The Log-Euclidean and affine-invariant arms test this directly. If they match Power-Cholesky, we will say the metric choice is a matter of convenience.

Why not just translate MRI into CT-looking images, or align features with a discriminator? That is SIFA and its 3D relative with shape-consistency losses (Zhang et al.). Translation models can invent or erase anatomy unless they are constrained by shape. Our input is meant to contain no appearance to translate. Plugging tensors into SIFA tests whether the two ideas are complementary.

Proposed Experiments

The primary experiment is MM-WHS 2017, in both directions (MRI→CT and CT→MRI), in two settings: target-free generalization and unsupervised adaptation. We report Dice and average surface distance per substructure. The challenge found the asymmetry that CT is easier, and the mechanism predicts larger gains in the harder direction and on myocardium, aorta, and pulmonary artery.

The volumetric-context experiment degrades both modalities to 3 mm and 5 mm through-plane spacing. We then compare resampling intensities first against resampling the tensor field with swelling-free interpolation. We also measure tensor determinant inflation and anisotropy loss at boundaries directly.

A secondary sanity check uses real diffusion tensors from the HCP data used by 3D TractFormer, where the reported tract Dice is 0.857. Rather than re-run tract segmentation, we verify that our swelling-free pyramid preserves fractional anisotropy across scales where Euclidean smoothing does not. This confirms that the averaging behaves as claimed on tensors with known physics.

The decisive checks are the following. The proposal succeeds if three things hold. First, the swelling-free field shrinks the CT-versus-MRI boundary angular-error gap relative to Euclidean smoothing. Second, that reduction carries over to Dice gains concentrated on thin structures. Third, the gains survive comparison with strong contrast augmentation.

The proposal fails cleanly if any of three things happen. The Euclidean ablation could match the swelling-free arm. Augmentation alone could match tensor input. Or the angular-error gap could shrink without any downstream segmentation gain, which would mean invariance at the input does not survive the network.