Large image encoders, such as the Segment Anything Model, are trained on flat photographs. When they are applied to a CT or MRI scan, they usually process the volume one slice at a time. This throws away information that runs through the depth axis. An organ boundary that is faint in one slice may be clear in its neighbours, but the per-slice encoder never sees them. The standard repairs are to smooth the stacked per-slice predictions afterwards or to train a 3D network on labelled volumes. The first is crude. The second needs labels and tends to be tied to the slice thickness it was trained on. This note proposes a third option. For each scan, with no labels or training, we recover which parts of a slice's features are shared with the slices above and below it. We then re-estimate that shared part from all three slices together.
The tool comes from causal discovery. With second-order statistics alone (covariances), a signal shared across slices cannot be pulled apart from its rotations. Third- and fourth-order statistics break that symmetry when the underlying sources are non-Gaussian. This is the same fact that lets linear non-Gaussian causal models identify causal direction and hidden confounders. We treat three consecutive slices as three "children" of hidden shared anatomical sources. The pixels of a single scan then supply the tens of thousands of samples that higher-order estimates need. Recovering these shared sources is, formally, the task of identifying a hidden confounder of three observed variables.
Core Claim
Across adjacent slices of one volume, the features a frozen 2D encoder produces can be modelled as a linear mixture: hidden non-Gaussian sources shared across the slices, plus parts specific to each slice. Under that model, the shared "depth context" is identifiable in closed form from a three-way cross-cumulant computed over in-plane pixels. A cumulant is a higher-order generalisation of covariance; the third-order one measures joint skewness. The claim has three parts. First, this per-scan, label-free fusion should improve slice-wise segmentation more than probability smoothing and more than a second-order shared-subspace method (canonical correlation analysis). Second, it should do so on thin structures such as the pancreas. Third, because the mixing is re-estimated for every scan, it should hold up when slice thickness at test time differs from training, which is where trained 3D fusion is expected to break.
The Model and Why Higher Order Is Needed
Let $F_{z}(p)\in\mathbb{R}^k$ be the encoder feature at in-plane location $p$ on slice $z$, after reducing to $k$ dimensions with a per-volume principal component projection. Within a local window of slices, we posit
$$F_{z+j}(p) = A_j\, s(p) + e_{z+j}(p), \qquad j\in\{-1,0,1\},$$
where $s(p)\in\mathbb{R}^m$ holds mutually independent non-Gaussian shared sources and $A_j$ is the $k\times m$ loading matrix for offset $j$. The residual $e_{z+j}(p)$ is the slice-specific part: its non-Gaussian component is independent across the three slices, while its Gaussian component may be correlated. Taking centred features $\bar F$ and averaging over pixels gives the third-order cross-cumulant tensor
$$\kappa = \mathbb{E}\big[\bar F_{z-1}\otimes \bar F_{z}\otimes \bar F_{z+1}\big] = \sum_{r=1}^{m} \gamma_r\, a_{-1,r}\otimes a_{0,r}\otimes a_{1,r},$$
where $\gamma_r$ is the third cumulant (skewness scale) of source $s_r$ and $a_{j,r}$ is column $r$ of $A_j$. Two useful properties follow. Slice-specific terms drop out because they are independent across the three slices. Any Gaussian component drops out whether or not it is shared, because Gaussian cumulants above order two are zero. As a result, the correlated slice-to-slice Gaussian component of reconstruction noise does not contaminate the estimate.
The tensor decomposition is unique under standard rank conditions. This uniqueness is exactly what second-order methods lack: canonical correlation analysis finds the shared subspace only up to rotation. Once the $A_j$ are known, we estimate $\hat s(p)$ from all three slices by least squares. We then replace the shared component of slice $z$ with $A_0\hat s(p)$, keep its slice-specific residual, and pass the result to the frozen decoder. Features after ReLU or GELU activations are typically skewed, which is what the third-order version needs. If skewness is weak, we fall back to the fourth-order cross-cumulant.
How This Relates to Prior Work
The identifiability logic is borrowed rather than invented. A Linear Non-Gaussian Acyclic Model for Causal Discovery established that non-Gaussian independent disturbances make linear structure identifiable where Gaussian ones do not. That is the core fact we rely on. Causal Discovery with Latent Confounders Based on Higher-Order Cumulants shows that cumulant ratios identify a latent confounder's loadings in closed form when the latent has at least three observed children. Our three-slice window is built to satisfy that condition. Triad Constraints for Learning Causal Structure of Latent Variables relaxes the requirement to two pure children using pseudo-residuals. We use its independence test as a diagnostic that the shared-source model fits a window. Learning Linear Bayesian Networks with Latent Variables shows that third-order moments untangle latent structure that variance cannot resolve, and it supplies the tensor-decomposition approach.
The file also flags the main practical constraint. The cumulant paper reports needing roughly 1000 to 2000 samples, which rules out estimating across patients in cohorts of a few dozen scans. That is why we estimate across pixels within a single scan instead.
The closest application of these methods to imaging is ParceLiNGAM: A causal ordering method robust against latent confounders. It orders simulated fMRI region-of-interest time series in the presence of a hidden region. However, it works on ten scalar signals and never touches voxel geometry. We apply the latent-source machinery to dense spatial feature maps instead.
We deliberately do not claim that slice $z$ causes slice $z+1$. The ordering of slices along the scan axis has no causal meaning, so "causal direction between slices" would be a category error. We use only the part of the theory that concerns latent-source identification.
On the imaging side, Deep convolutional networks for pancreas segmentation in CT imaging documents the depth-context problem directly and applies isotropic 3D Gaussian smoothing to stacked 2D probabilities. We replace that fixed, symmetric blur with a scan-specific mixing estimated from data, applied to features rather than probabilities. Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation argues that clinical anisotropy (fine in-plane resolution, thick slices) makes isotropic 3D kernels wasteful. It separates 2D in-plane from 3D cross-slice processing with learned gates. We share that decomposition, but our cross-slice step is estimated per scan without labels rather than learned once. nnFormer: Interleaved Transformer for Volumetric Segmentation and TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation are fully trained volumetric systems. We treat them as upper references in distribution, not as methods we expect to beat there.
Research Program
The work has four components. The first is a direct test of the modelling assumptions on real encoders. For a frozen 2D encoder and for a 2D U-Net trained slice-wise, we measure the skewness and kurtosis of reduced features per scan. We apply the Triad independence test to check whether residuals behave as the shared-source model predicts. We also vary the window size and the number of retained dimensions.
The second is a controlled recovery study. We start from thin-slice CT, synthesise thick slices by averaging over known slice profiles, and check whether the recovered loadings $A_j$ track the known profile and overlap. We plot loading error against the number of pixels pooled, to find where the sample-size limit from the cumulant paper binds.
The third is the fusion module itself. It is a closed-form, per-window computation: a principal component projection, estimation of the cross-cumulant tensor, a tensor decomposition, and least-squares source estimation. It is implemented on top of the IMOP-lab/U-Shaped-Connection and Beckschen/TransUNet codebases so that the encoders and evaluation pipelines are shared. Because pixels are spatially autocorrelated, the effective sample size is smaller than the pixel count. We quantify the resulting estimator variance with a spatial block bootstrap, and we down-weight windows whose decomposition is unstable.
The fourth is the evaluation under slice-thickness shift described below.
Baselines
The naive reference point is plain slice-wise inference: the same frozen or slice-trained 2D encoder and decoder applied to each slice independently, with no cross-slice step. Every other method must improve on this. For slice-wise inference with a frozen 2D foundation model on Synapse or BTCV, no reported result was found on Lacuna, so we measure it ourselves.
The first substantive baseline is 3D Gaussian smoothing of the stacked per-slice probabilities, following Deep convolutional networks for pancreas segmentation in CT imaging. That paper reports that this step raised pancreas Dice from 56% to 68% (±10%) on its 82-volume NIH pancreas CT set. It is the strongest label-free competitor, and it is essentially free to run. We will run it on the same slice-wise outputs, with the kernel width tuned on the same validation split and search budget given to our window size. We also run a feature-level version that smooths encoder features along depth, so that the comparison isolates the choice of fusion rule rather than whether fusion happens on features or probabilities.
The ablation that isolates the key ingredient is a second-order version of our own module. It uses canonical correlation analysis across the three slices to find the shared subspace, then applies the same least-squares re-estimation and the same decoder. The only thing it lacks is the higher-order decomposition that resolves the rotation. If it matches us, then non-Gaussianity is not doing the work, and the causal-identifiability framing is decoration. We also run our module with sources forced to $m$ equal to the canonical correlation rank, so the two methods operate at matched capacity.
The strongest existing methods are trained volumetric models, specifically the uC 3DU-Net and 3D TransUNet. For reference, TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation reports 77.48% average Dice on Synapse, and the Beckschen/TransUNet README reports 88.11% Dice for 3D TransUNet on BTCV. For the uC 3DU-Net, the file reports only relative gains on FLARE2021 over 3D UX-Net, including a 3.1% pancreas Dice increase, and no absolute value. We retrain these models on identical splits.
We also add a trained lightweight 3D fusion head on top of the same frozen 2D features, with equal training data. This measures what labels buy relative to label-free fusion. In distribution, beating the trained models is not the goal. Our module counts as successful there if it closes a substantial fraction of the gap between slice-wise inference and trained 3D fusion while beating both smoothing and canonical correlation analysis. Under slice-thickness shift, success means matching or exceeding the trained models.
Why not just…?
Why not just smooth the per-slice predictions, as Roth et al. did? Smoothing is the right first move and it is our first baseline. However, it applies the same symmetric blur regardless of slice thickness or of how much each feature actually persists across slices. It therefore erodes thin and faint structures along with the noise. Roth et al.'s own 68% pancreas result shows that there is headroom beyond smoothing. The decisive comparison is on pancreas and small-organ Dice and on 95th-percentile Hausdorff distance, together with a measure of how often labels flip between neighbouring slices. If feature-level smoothing matches us on all of these, the higher-order machinery is not needed.
Why not just use a second-order shared-subspace method such as canonical correlation analysis? This is the most serious threat. Canonical correlation analysis already finds what adjacent slices share, and it is cheaper and more stable at small sample sizes. Our argument is that within the shared subspace, second-order methods cannot separate anatomical structure from correlated Gaussian-like noise and artifacts, whereas the cross-cumulant ignores Gaussian components by construction. The matched-capacity ablation settles this directly. We expect our advantage to be largest on low-dose or thick-slice scans, where correlated noise is strongest, and to shrink on clean thin-slice data. If it does not appear even there, we will report that the second-order method is sufficient.
Why not just train a small 3D adapter on top of the frozen 2D features? With enough labelled volumes at matching slice thickness, a trained adapter will probably win in distribution. The anisotropy analysis in Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation suggests why it may not transfer: cross-slice relationships depend on slice spacing, and a learned fusion encodes whatever spacing it was trained on. The slice-thickness shift experiment tests exactly this. We also report a learning curve in the number of labelled volumes, to show where label-free fusion stops being competitive.
Proposed Experiments
In-distribution segmentation uses Synapse (8 organs), BTCV (13 organs), FLARE2021, and the NIH pancreas set. We report Dice and 95th-percentile Hausdorff distance, plus an inter-slice consistency score: the fraction of surface points whose label changes between adjacent slices where the ground truth does not change. ACDC cardiac MRI, which has very thick slices, serves as a natural high-anisotropy test.
The slice-thickness shift experiment trains every learned method on thin-slice reconstructions. It then tests on the same scans resampled to 3 mm and 5 mm thickness using realistic slice profiles, and additionally trains thick and tests thin. The thickness range follows the in-plane versus out-of-plane spacing gap described in the file.
The decisive checks are three. Recovered loadings on synthetic thick slices must track the known slice profile better than a canonical-correlation rotation, with error falling as more pixels are pooled. Encoder features must be measurably non-Gaussian; if skewness and kurtosis are near zero, the premise fails and we stop. And under thickness shift, our module must degrade less than the trained 3D fusion head. If the second and third checks both fail, the idea is wrong for this application, and we will report the negative result.
References
- A Linear Non-Gaussian Acyclic Model for Causal Discovery
- Causal Discovery with Latent Confounders Based on Higher-Order Cumulants
- Triad Constraints for Learning Causal Structure of Latent Variables
- Learning Linear Bayesian Networks with Latent Variables
- ParceLiNGAM: A Causal Ordering Method Robust Against Latent Confounders
- Deep convolutional networks for pancreas segmentation in CT imaging
- Upping the Game: How 2D U-Net Skip Connections Flip 3D Segmentation
- nnFormer: Interleaved Transformer for Volumetric Segmentation
- TransUNet: Transformers Make Strong Encoders for Medical Image Segmentation