When a CT scanner takes few or noisy X-ray projections, the reconstructed image has to be inferred, and the network doing that inference makes choices about what to show. Training that network jointly with a downstream nodule segmenter helps the segmenter a great deal, but there is a known cost: the reconstruction becomes visibly worse overall. The network learns to distort the image wherever doing so helps the segmenter. We propose to confine that distortion to the places where the segmenter actually draws its evidence. During training, the reconstruction network is penalized for deviating from the true image everywhere except where the gradient of the segmentation loss with respect to the image is large. The result is a single image that stays faithful to anatomy for a human reader across most of the field of view and is tuned for the machine only in a small, explicitly mapped region.
This matters for tiny targets in particular. Lung nodules of a few millimetres are exactly the structures that both reconstruction smoothing and network downsampling erase first, so they are where task adaptation pays off most and where it is most dangerous to hide. The proposal uses gradient saliency (a heatmap of how much the model's output changes when each input pixel changes) as an active training signal, not as a picture inspected afterwards. It also produces, as a by-product, an audit map showing where a deployed image was shaped for the algorithm.
Core Claim
Empirical evidence of the task-adapted reconstruction framework for joint CT reconstruction and segmentation trains a reconstruction U-Net and a nodule-segmentation U-Net together on LIDC-IDRI using a global mix $C\cdot L_{seg} + (1-C)\cdot L_{rec}$. In the noisy 64-view setting, nodule Dice rose from 0.012 for the sequential pipeline to 0.214. The joint model achieved this partly by lowering reconstruction PSNR, which the authors call the PSNR paradox. They leave open how to get the segmentation gain without splitting the output into one image for clinicians and another for the algorithm.
Our claim is that most of that PSNR loss is wasted, because the segmenter needs only a small fraction of the image to change. We test it by localizing the trade-off spatially with a stop-gradient saliency weight:
$$L = L_{seg}(f(\hat x), y) + \mu \sum_i \big(1 - w_i\big)\,(\hat x_i - x_i)^2, \qquad w = \mathrm{norm}\big(G_\sigma * |\nabla_{\hat x} L_{seg}|\big).$$
The symbols are as follows. $\hat x$ is the reconstruction from an unrolled network. $x$ is the reference full-dose image. $f$ is the segmenter and $y$ the nodule mask. $L_{seg}$ is a hybrid Dice-plus-focal loss. $\mu$ sets the fidelity strength. $w_i\in[0,1]$ is the saliency at pixel $i$, computed by averaging input gradients over a few noise-perturbed copies of $\hat x$ (the SmoothGrad approach used in Deep learning interpretation of echocardiograms), then blurring with a Gaussian $G_\sigma$ and normalizing to a fixed area budget.
The prediction is a better Pareto front, meaning nodule Dice can be raised without the usual drop in background PSNR. Compared with the global-$C$ joint model, at matched nodule Dice we expect clearly higher PSNR outside a dilated nodule region.
There is also a physics prediction. Because each unrolled stage enforces consistency with the measured sinogram, the edits the segmenter can buy should lie mostly in the null space of the sparse-view projector, the image components the 64 measurements cannot see. We will measure this directly.
Research Program
The reconstruction network is an unrolled proximal forward-backward splitting network. Unrolling means a fixed number of iterations of a classical solver, each pairing an exact data-consistency step through the CT projector with a learned denoising block. We follow Low-dose CT with deep learning regularization via proximal forward–backward splitting, which supplies the method and reports 44.04 dB on its own prostate low-dose data. We use about ten stages. Unrolling is chosen over the plain U-Net of Valat et al. so that measurement consistency is enforced by construction, which makes the null-space analysis meaningful.
The segmenter is a 2D/2.5D U-Net trained with Dice plus focal loss. That hybrid is standard practice, shown for small or imbalanced targets in Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation and Deeply supervised network for white matter hyperintensities segmentation with transfer learning. The loss is held fixed; it is not an object of study here.
We compute saliency at image resolution, as an input gradient, rather than with Grad-CAM. Grad-CAM maps live at the coarse bottleneck and would smear across exactly the millimetre-scale structures at stake. This is the coarseness problem noted for the post-hoc Grad-CAM use in Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets. Grad-CAM-derived weights remain as an ablation.
At test time no labels exist, so the deployed reconstruction is simply $\hat x$. We additionally release the saliency of the predicted foreground score as an audit map. We will check that this label-free map agrees with the training-time map.
The analysis has three parts. First, we decompose $\hat x - x$ into the range of the projector's adjoint and its orthogonal complement (the null space), separately inside and outside high-saliency regions. Second, we track how saliency mass shifts across training: does it sit on nodules, on vessels that cause false positives, or on reconstruction streaks? Third, we test spatial-resolution degradation inside the segmenter. We swap its max-pooling for learned importance pooling, following LIP: Local Importance-Based Pooling, which raised COCO small-object AP from 21.9% to 24.0%. This tells us whether nodule loss happens in reconstruction or in the segmenter's own downsampling.
Baselines
The naive reference is the sequential pipeline. The unrolled network is trained for PSNR alone and frozen, and the same segmenter is then trained on its outputs. Valat et al. report Dice 0.012 for their sequential model in the noisy 64-view setting. We reproduce our own version on the same splits rather than borrowing the number.
The strongest existing method is the global-weight joint model of Valat et al., which reports nodule Dice 0.214 in noisy sparse view. No reported PSNR value for it was found on Lacuna, so we will produce its Dice-versus-PSNR curve ourselves. Fairness is handled the same way for every model: the same unrolled reconstructor, same segmenter, same LIDC-IDRI simulation geometry, same training epochs, and the same hyperparameter search budget. That budget sweeps $C$ for the joint baseline and $\mu$ and the area budget for ours. Beating it means a curve that lies strictly above theirs. Concretely, at their best Dice, our background PSNR should be higher by more than the seed-to-seed spread. Alternatively, at equal background PSNR, our Dice should be higher.
The key ablation is mask-weighted fidelity. Here $w$ is replaced by the dilated ground-truth nodule mask, normalized to the same area. This isolates whether gradient saliency adds anything beyond "relax fidelity on the lesion." We also run three further controls. One uses a random weight map of equal area. One uses Grad-CAM weights. One is a dual-output model that emits a separate image for the segmenter; it is an upper reference for segmentation when fidelity is completely unconstrained.
On real-clinic reconstructions, S4ND: Single-Shot Single-Scale Lung Nodule Detection reports CPM 0.897 on LUNA16. We use it only as a ceiling for detection from full-dose images.
Why not just…?
Why not just down-weight fidelity inside the ground-truth nodule mask? This is the most serious threat, and it is why mask-weighting is the central ablation. Our reason to expect saliency to do better is that segmentation failures in sparse-view CT include false positives. Streaks and vessel cross-sections can look like nodules, and those lie outside any ground-truth mask. The saliency of the segmentation loss lights up there, but a mask cannot. The decisive measurement is therefore precision and false positives per scan, not only Dice. If mask-weighting matches saliency on both, we will report that the simpler method suffices and that saliency's value is limited to the test-time audit map.
Why not just output two images, one for the clinician and one for the algorithm? This would very likely reach the best segmentation, and we include it as an upper reference. Its cost is the one Valat et al. flag: the algorithm's evidence is then an image no human reads, so its decisions cannot be checked against what the radiologist sees. Our question is how much of the dual-output Dice can be recovered inside one shared image. We report that recovered fraction explicitly.
Why not just fix the segmenter's downsampling with learned pooling and keep the sequential pipeline? If nodules are lost mainly because pooling dilutes them, LIP-style pooling alone could close much of the gap. Running sequential-plus-LIP separates the two failure sources. If it approaches joint-model Dice, then the reconstruction-side story is weaker than claimed. Valat et al.'s near-zero sequential Dice suggests the information is largely gone before the segmenter sees the image, but that has not been tested with detail-preserving pooling.
Proposed Experiments
The primary benchmark is LIDC-IDRI with simulated sinograms, reproducing Valat et al.'s four regimes: noisy and noiseless, each at 64 and 1024 views. For every method we sweep the trade-off parameter and plot nodule Dice against PSNR computed outside a 5 mm dilated nodule region. We also report PSNR inside that region, sensitivity, precision, false positives per scan, and the sinogram residual.
The first decisive check is the Pareto comparison against global-$C$ joint training in the noisy 64-view regime. The second is the mask-weighting ablation, read on false positives. The third is the null-space analysis: if more than most of the task-induced deviation lies in the null space, the edits are "invisible" to the measurements, which bears directly on whether they should count as hallucination.
To test fidelity of the audit map, we perturb the top-saliency pixels of the test reconstruction and require segmentation output to change much more than when perturbing random pixels of equal area. As a secondary transfer test, we apply the same loss to accelerated knee MRI on SKM-TEA. There, the transformer joint framework of Transformer-Based Integrated Framework for Joint Reconstruction and Segmentation in Accelerated Knee MRI reports PSNR 35.550 dB and Dice 0.825. Its ablation found that a spatially uniform segmentation-feedback term traded segmentation accuracy for reconstruction quality, which is exactly the tension saliency localization targets. The proposal fails if, in the noisy sparse-view regime, the saliency-weighted front does not separate from both the global-$C$ front and the mask-weighted front beyond seed variance.
References
- Empirical evidence of the task-adapted reconstruction framework for joint CT reconstruction and segmentation
- Deep learning interpretation of echocardiograms
- Low-dose CT with deep learning regularization via proximal forward–backward splitting
- Enhanced U-Net: A Feature Enhancement Network for Polyp Segmentation
- Deeply supervised network for white matter hyperintensities segmentation with transfer learning
- Artificial intelligence for the detection of COVID-19 pneumonia on chest CT using multinational datasets
- LIP: Local Importance-Based Pooling
- S4ND: Single-Shot Single-Scale Lung Nodule Detection
- Transformer-Based Integrated Framework for Joint Reconstruction and Segmentation in Accelerated Knee MRI