Tuning-Free Structure–Texture Separation for Anisotropic 3D Scans

Many image-processing pipelines first split an image into two layers. One is a "cartoon" layer of smooth regions and sharp boundaries. The other is a "texture" layer of fine, repeating, or noisy oscillation. The split is useful because smoothing and sharpening operators can then act on each layer without damaging the other. Every method for doing this split depends on a regularization weight that sets the scale at which an intensity change counts as an edge rather than an oscillation. Recent work learns that weight for 2D images. Medical scans and other volumes, however, have a property that 2D methods never face: their voxels are anisotropic, meaning they are spaced differently along different axes. A CT or MRI volume often has 0.7 mm in-plane spacing and 3–5 mm between slices, and the ratio changes from scanner to scanner and protocol to protocol. So the boundary between edge and oscillation differs along the slice axis and within the plane, and it shifts with every new acquisition. That is a hyperparameter problem a 2D method cannot even represent.

We propose a decomposition network built by unrolling an optimization algorithm, meaning a fixed number of solver iterations is turned into network layers. The network separates a 3D volume into structure and texture with no per-dataset tuning. It does this through two choices. First, the texture prior is a low-rank-patch model applied on the three orthogonal slicing planes and coupled through a single shared structure volume, which avoids the memory cost of 3D patch matrices. Second, the edge-scale weights are predicted separately for each axis from the volume's voxel spacing and local statistics, instead of being fixed scalars. The central claim is testable. One network trained once on synthetic data should decompose volumes of unseen anisotropy and noise level as well as classical solvers given oracle-tuned weights. It should also beat both slice-by-slice 2D decomposition and decomposition after resampling to isotropic voxels.

What exists and what is missing

The closest work is LPR-NET, from Parameter-free structure-texture image decomposition by unrolling. It turns the alternating direction method of multipliers (ADMM), a solver that alternates between subproblems, into a fixed-depth network. In that model, texture is the part whose stacked image patches form a low-rank matrix, enforced by a singular value decomposition (SVD) step. Structure is regularized by the minimax concave penalty (MCP), a non-convex edge penalty that avoids the staircase artifacts of total variation. LPR-NET learns its balancing weight from synthetic data and thereby removes manual tuning. We reuse its algorithmic skeleton. It is entirely 2D, however: it uses 2D patches and 2×2 filters, and it learns one global trade-off rather than one per axis. Extending its patch SVD naively to 3D patches would multiply matrix sizes cubically.

Two papers motivate the problem. Bregman implementation of Meyer's G-norm makes Meyer's oscillatory-texture model fast through Split Bregman iterations. It still fixes its weights at λ = μ = 1000, and the authors name the lack of adaptive parameters as unresolved. Adaptive Texture Filtering for Single-Domain Generalized Segmentation shows that predicting a per-image filtering strength from simple intensity statistics (mean and variance) helps segmentation generalize across domains. Its predictor outputs one scalar for a 2D driving image, and it knows nothing about voxel geometry.

On the volumetric side, two papers supply methods we borrow. Tri-Plane Mamba shows that scanning three orthogonal planes gives 3D context cheaply, at 2.37 versus 18.86 GFlops per block against 3D self-attention. It uses tri-planes for feature context and does not separate structure from texture. We instead use tri-planes as the support of a texture prior. Efficient Unrolled Networks for Large-Scale 3D Inverse Problems shows that unrolled 3D solvers out-of-memory on 80 GB GPUs at 501³ voxels unless the volume is partitioned into patches. We adopt its partitioning for full-resolution inference. Its task is reconstruction, not decomposition.

The AMOS benchmark supplies the downstream stress test. It reports a drop of more than 20% when models meet MRI from unseen scanners, which the authors attribute to soft-tissue appearance shifts.

Core Claim

We write an observed volume as $f = u + v$, where $u$ is the structure volume and $v$ is the texture volume. The unrolled network approximately solves

$$\min_{u,v}\ \sum_{a\in\{x,y,z\}} \mu_a\, \Phi_{\mathrm{MCP}}(\partial_a u) \;+\; \sum_{o\in\{xy,xz,yz\}} \gamma_o \sum_{s} \big\|\mathcal{P}(v_{o,s})\big\|_* \quad \text{s.t. } f = u+v.$$

The symbols are as follows. $\partial_a$ is the finite difference along axis $a$ scaled by that axis's voxel spacing. $\Phi_{\mathrm{MCP}}$ is the minimax concave penalty summed over voxels. $v_{o,s}$ is slice $s$ of $v$ taken in plane orientation $o$. $\mathcal{P}$ stacks the 2D patches of a slice into a matrix. $\|\cdot\|_*$ is the nuclear norm, the sum of singular values, which is small when the patches repeat.

The weights $\mu_a$ and $\gamma_o$ are not hyperparameters. A small predictor $g$ outputs them at every unrolled stage, voxel-wise, from the spacing vector and local statistics of the current residual. The whole network is trained end to end on synthetic volumes.

Two things are new. The per-axis edge weights capture anisotropy that a single learned scalar cannot. The plane-wise low-rank texture term costs roughly three times a 2D SVD pass per slice instead of one cubic 3D patch matrix. The shared $u$ couples the three planes, so texture cannot be "explained" differently in each orientation.

Research Program

The first deliverable is a synthetic 3D benchmark with exact ground truth. LPR-NET's 2D generator builds smooth random shapes with the Lane–Riesenfeld curve-subdivision algorithm and textures from sparse Fourier components. We lift it to 3D: smooth closed surfaces built by subdivision, oriented 3D sparse-Fourier textures, and additive noise. We then simulate acquisition by anisotropic blurring and downsampling along one axis, with slice-to-pixel spacing ratios from 1 to 6. Training uses a sub-range of ratios and noise levels. Testing holds out both the extremes and an interior band. That separates interpolation from extrapolation, which is exactly where fixed hyperparameters break.

The second deliverable is the network itself, with depth K between 5 and 15 stages. Inference on clinical-size volumes uses the target/context partitioning of Vo and Tachella. Their trick is simpler here because the decomposition's forward operator is the identity, so context only enters through the finite differences and patches at block borders.

The third deliverable is a sensitivity analysis, which is the point of the hyperparameter direction. For every method we report two numbers. One is the oracle score, with weights tuned per test condition. The other is the transferred score, with weights tuned once on the training conditions. The gap between them is the method's tuning dependence, and our claim is that this gap is near zero for the proposed network. The network's own remaining design choices are depth, patch size, and the training spacing range, and we sweep them to show they are not hidden tuning knobs.

Baselines

The naive reference is classical 3D TV structure–texture decomposition. It uses a single weight tuned by grid search on the training-condition validation set and is then applied unchanged. This is what a practitioner would do today. Its oracle variant, retuned per test condition, is an upper bound for any fixed-weight method, and it is the comparison that makes our tuning-free claim meaningful. If the proposed network is within 0.3 dB PSNR of oracle-tuned 3D TV, and clearly above transferred 3D TV, on held-out anisotropy, that counts as success. All classical baselines get the same tuning budget in GPU-hours that our network uses for training.

The strongest existing method is LPR-NET, run in two ways. The first runs it slice by slice along the acquisition axis and stacks the results. The second runs it on all three plane orientations independently and averages, which is the obvious "tri-plane" shortcut. Both are retrained on 2D slices drawn from our synthetic volumes, using the same data and steps. The only reported anchor is on the original 2D 64×64 benchmark. There LPR-NET reaches 42.13 dB PSNR, the iterative ADMM low-patch-rank solver 41.61 dB, TV-L2 38.84 dB, and heavier plug-and-play models 42.69 dB (LPR-NET paper). On 3D or anisotropic data there is no reported result found on Lacuna for LPR-NET or any decomposition method. Our first sanity check is therefore to restrict our network to isotropic 2D and confirm it reproduces about 42.1 dB on that benchmark before any 3D claim is made.

A third baseline isolates the adaptive-weight idea. It ports the adaptive strength predictor of Li et al. into our unrolled network: one scalar weight per volume, predicted from mean and variance and blind to spacing. Two ablations isolate our key ingredients. One shares the three axis weights, so $\mu_x=\mu_y=\mu_z$ and anisotropy can only enter through scaled differences. The other replaces the coupled plane-wise texture term with a single-orientation term. The claim requires both ablations to lose specifically at high slice-spacing ratios.

Why not just…?

Why not just resample every volume to isotropic voxels and use an isotropic decomposition? This is the most serious alternative. Interpolating along the slice axis does not create the missing high frequencies. It produces smooth, blurred slices in which the slice-direction edge scale is still set by acquisition, so a single isotropic weight would still be wrong along that axis. Resampling also inflates volume size by the spacing ratio. Resampling is cheap, though, and it might work. We therefore run it as a full baseline: resample, then apply both 3D TV and isotropic-trained versions of our network. If it matches the anisotropic-weight network on held-out ratios, the per-axis mechanism is unnecessary, and we would report that and reduce the claim to the tri-plane texture prior.

Why not just set the weight from a noise estimate, a classical heuristic rule? A rule that scales λ with an estimated noise level handles noise but not anisotropy or texture scale. The Gilles–Osher solver's sensitivity at fixed λ = μ = 1000 is about texture scale as much as noise. We include such a rule-based TV baseline. The decisive cell is fixed noise with varying spacing ratio, where a noise-only rule cannot adapt.

Why not skip decomposition and fix scanner shift with augmentation? For segmentation this is a real competitor, and AMOS shows only that standard pipelines still lose more than 20% on unseen MRI scanners. AMOS does not show that decomposition fixes this. The 2D evidence of Li et al. (44.24% mIoU, about 2 points over prior methods) favors explicit texture suppression, but in a very different domain. We treat the downstream experiment below as genuinely uncertain. The core contribution does not depend on it.

Proposed Experiments

The primary experiment is the synthetic 3D benchmark. We report PSNR of $u$ and $v$ against ground truth, broken down by spacing ratio and noise level, and separately for interpolation and extrapolation conditions. We also report the oracle-minus-transferred gap for every method and peak GPU memory.

The second experiment uses real volumes, where no ground truth exists. It is a reslicing-consistency check. On AMOS CT volumes with thin slices, we simulate thick-slice acquisitions by averaging adjacent slices. We then measure how closely the structure layer from the thick-slice version matches the downsampled structure layer of the thin-slice original. A tuning-free method should keep this agreement stable across simulated slice thickness, while fixed-weight methods should drift.

The third experiment is downstream segmentation on AMOS. We train the 3D-UNet, the strongest architecture there at 89.04% CT mDice (AMOS), on CT plus MRI from a subset of scanners and test on MRI from held-out scanners. There are three inputs: raw intensity with standard augmentation, raw plus the 3D-TV structure channel, and raw plus our structure and texture channels. Augmentation and training budget are identical across the three. We additionally report per-organ Dice for the adrenal glands and duodenum, the thin structures AMOS flags as hardest.

Success is a measurable reduction in the unseen-scanner drop without loss on in-distribution CT. A null result here would be informative and reported as such. The decisive checks for the core claim stay in the first two experiments: matching oracle-tuned baselines on held-out anisotropy with zero tuning, and both ablations failing specifically where slice spacing is coarse.