Learning 3D Tumor Detectors from What Radiology Reports Say Is Absent

Most CT radiology reports say that something is not there. "No focal hepatic lesion," "pancreas unremarkable," "kidneys without mass": these sentences outnumber positive tumor descriptions by a wide margin, because healthy organs far outnumber diseased ones. Current methods that learn 3D segmentation from reports throw this signal away or treat it as background. They also discard positive findings written without a measurement ("ill-defined hypodensity in the pancreatic tail"). We propose to turn each report into a set of organ-level statements, each marked present-with-size, present-without-size, absent, or hedged. Each statement then becomes a constraint on the 3D tumor probability map inside that organ. An "absent" statement forces the highest tumor probability anywhere in the organ to stay low. A size-less "present" statement forces at least a small region of the organ to light up. Sized findings keep the existing diameter-based losses.

The bet is that one mechanism addresses two problems. The class imbalance that makes tumor models flood clinicians with false positives can become supervision, because the abundant negative reports are free hard-negative examples. The rare, irregular, often unmeasured findings in the long tail can also start contributing gradient. The constraints act on the full 3D volume rather than on 2D slices. The text extraction step, normally a manual chart-review bottleneck, is done by a language model, and we explicitly audit and model its errors.

Core Claim

Learning Segmentation from Radiology Reports (R-Super) is the closest work and supplies our method base. It uses Llama 3.1 70B to extract tumor count, organ, and diameter from 6,718 UCSF CT reports. It then trains a 3D segmenter (MedFormer) with two losses. A Volume Loss matches the predicted volume to an ellipsoid built from the reported diameter. A Ball Loss slides a diameter-sized 3D ball over the probability map and rewards probability inside the best-matching ball. Both losses need a number. The research file records that R-Super cannot learn from size-less mentions, assumes near-spherical tumors, and does not address imbalance between routine and rare scans.

AbdomenAtlas-8K motivates the proposal from the other side. Its active-learning loop labeled 8,448 CT volumes for organs with a 533x speedup. The authors report that tumors broke this loop, because models produced excessive false positives under extreme imbalance. They also had no reports with which to filter those false positives.

Our claim is that organ-level existence constraints, derived from negated and size-less report statements, lower false-positive tumor detections on healthy organs. They should also raise detection of rare tumor types relative to diameter-only report supervision. The gains should concentrate exactly where R-Super's losses are silent: reports with no measurement, and organs reported as normal. The file does not say whether R-Super uses tumor-free reports as explicit negatives. Our ablations therefore isolate the negative constraint rather than assume it is new by construction.

Research Program

The first component is extraction. A language model reads each report and emits, for every organ in a fixed abdominal list and every finding type, one of four states plus an extraction confidence. The four states are present-with-size, present-without-size, absent, and hedged ("cannot exclude," "possible").

Negation and hedging are the documented weak point of language-model extraction. We therefore hand-audit a stratified sample of several hundred reports and measure state-level accuracy separately for each state. We also deliberately inject state-flip noise at controlled rates to measure how extraction errors propagate into the segmenter. This audit is a deliverable in its own right. It tells practitioners how much reading time a radiologist still has to spend.

The second component is the loss. Let $p(v)$ be the predicted tumor probability at voxel $v$. Let $O_k$ be the mask of organ $k$ from an off-the-shelf organ segmenter, as in R-Super. Let $m_k = \mathrm{mean}_{\text{top-}K}\{p(v): v \in O_k\}$ be the mean of the $K$ largest probabilities in the organ. This is a smoothed maximum, and $K$ is set to roughly the voxel count of the smallest clinically relevant lesion. The existence loss for statement $s$ about organ $k$ is

$$ \mathcal{L}_{s} = w_s \, \mathrm{BCE}\big(m_k,\; y_s\big), \qquad w_s = c_s \cdot f_{t(s)}^{-\alpha}, $$

The symbols are defined as follows:

  • $y_s$ is 0 for "absent" and 1 for "present without size."
  • $c_s$ is the extraction confidence.
  • $f_{t(s)}$ is the corpus frequency of the finding type $t(s)$ among positive statements.
  • $\alpha \in [0,1]$ controls long-tail reweighting.

Hedged statements receive $w_s = 0$ in the main model; a variant uses a soft target instead. Sized findings additionally receive R-Super's Ball and Volume losses. Any available voxel masks receive ordinary Dice and cross-entropy.

Because $m_k$ is computed over the whole 3D organ rather than per slice, a single false-positive blob on any slice is penalized. A real lesion spanning only two of 300 slices still satisfies the positive constraint. This is the volumetric point that slice-level attention pooling, as in Fair Lung Disease Diagnosis from Chest CT via Gender-Adversarial Attention Multiple Instance Learning, cannot exploit.

The third component is analysis. We will report results separately by report type (sized positive, size-less positive, negative), by tumor-type frequency tier, and by organ. The aim is to show where any gain comes from, not just that an average moved.

Baselines

The naive reference point is a 3D segmenter with the MedFormer architecture trained only on whatever manual voxel masks are available. It uses ordinary Dice plus cross-entropy and treats report-negative scans as all-background. This baseline shows how much any report supervision helps at each mask budget.

The strongest existing method is R-Super with Ball and Volume losses. The cited paper reports that it raises tumor-detection F1 "by up to 16% in low-data regimes and 4.3% when large amounts of manual masks were already available." It also reports outperforming multi-task and CLIP-style baselines. The research file gives no absolute Dice or F1 for R-Super on AbdomenAtlas 2.0 or JHH-Test, so we will reproduce it ourselves. We will use the same architecture, the same extracted reports, the same organ masks, the same mask budgets, and the same number of training steps and hyperparameter trials as our method. Beating it means two things at matched voxel-level sensitivity. Per-lesion detection F1 on the external JHH-Test must improve by more than the across-seed standard deviation. False-positive components per report-negative scan must drop.

R-Super plus focal loss with class-balanced sampling is a separate baseline. The reweighting follows Topology-Aware Focal Loss for 3D Image Segmentation and AdaFocal, and is tuned with the same budget. It tests whether generic imbalance handling already captures the gain.

R-Super plus an auxiliary per-organ classification head is the structurally closest simple competitor. The head is trained on the same four-state labels, with absent mapped to 0 and size-less present to 1, but it is not tied to the voxel map.

Ablations isolate the key ingredient. We will run our method with only negative constraints, with only size-less positive constraints, with $\alpha = 0$ (no long-tail reweighting), and with $c_s = 1$ (no confidence weighting). We will also run a version that uses a per-slice maximum averaged over slices instead of the organ-wide 3D top-$K$ mean.

Why not just…?

Why not just add focal loss and oversampling to R-Super? Voxel-level reweighting tells the model that tumor voxels matter more. It cannot tell the model that this particular liver contains no tumor at all, which is the information needed to suppress a confident false-positive blob. The research file's evidence is that this recipe plateaus under combined sparsity and scarcity. The Fair Lung paper used focal loss, weighted sampling, and threshold tuning and still reached only 0.366 F1 on its rarest class. That evidence comes from a classification task, though, so it is suggestive rather than decisive. The focal-loss baseline above settles it directly on the same data.

Why not just train an auxiliary organ-level classifier on the extracted labels? This is the real threat, because it uses identical text-derived information with less machinery. Our argument is that an auxiliary head can learn "liver is normal" while the segmentation map still contains spurious tumor voxels. Our constraint, by contrast, acts on the map itself. If the auxiliary-head baseline matches our false-positive rate and detection F1, the contribution reduces to "use negative and size-less labels at all," and we will report that plainly. As a diagnostic, we will measure agreement between the auxiliary head's organ prediction and the presence of predicted tumor components in that organ. Disagreement would show the gap our loss is meant to close.

Why not just filter predictions with the report at annotation time? In a pseudo-labeling pipeline like AbdomenAtlas-8K's, reports exist at labeling time. Deleting tumor predictions in report-negative organs is trivial and would reduce reviewer load. It does nothing when no report exists yet, which is the screening and deployment case, and it does not improve the model's own representation. We will include report-gated filtering as an annotation-workload reference. Our method must reduce false positives on scans whose reports are withheld at test time.

Why not just rely on a contrastively trained 3D vision-language model? CT-CLIP aligns 3D CT volumes with free-text reports. It reports "+0.102 higher mean AUROC" over supervised CT-Net internally and "+0.085 AUROC" on external RAD-ChestCT. The file also notes that its accuracy drops substantially on rare pathologies and is sensitive to prompt phrasing. It produces no voxel-level output either, so it cannot localize findings or support the annotation workflow. We include it on the chest benchmark below.

Proposed Experiments

The primary setting reproduces R-Super's abdominal benchmark. Training uses the report-paired UCSF cohort, or the authors' released split if access permits. Internal testing uses AbdomenAtlas 2.0 and AbdomenAtlas-Small, and external testing uses JHH-Test. We vary the number of manual masks from near zero to the full set. The decisive measurements are three, each reported with seeds and confidence intervals:

  • per-lesion detection F1 on JHH-Test;
  • false-positive tumor components per scan on scans whose ground truth contains no tumor;
  • sensitivity on tumor types in the bottom frequency tier.

The first decisive check is the error-locating test. Gains over R-Super should appear mainly on size-less positives and report-negative organs, and be near zero on sized positives where both methods receive the same supervision. A uniform gain would suggest a regularization effect rather than our mechanism.

The second check is extraction robustness. Performance should degrade gracefully as injected negation flips rise from 0 to 20 percent. The confidence weighting should slow that degradation relative to $c_s = 1$.

The third check is volumetric. The organ-wide 3D top-$K$ mean should beat the slice-averaged variant on false positives. If it does not, the volumetric framing is unnecessary.

A secondary chest experiment tests transfer. We train on CT-RATE reports with lung-lobe masks and score the 18 extracted abnormalities through each lobe's top-$K$ probability. Internal AUROC is compared against CT-CLIP, and external AUROC on RAD-ChestCT. The file reports CT-CLIP only as differences from CT-Net, not as absolute AUROC: no reported absolute result found on Lacuna for CT-CLIP's mean AUROC. We therefore rerun CT-CLIP's released weights on our split.

Finally, we estimate the annotation benefit. We simulate an AbdomenAtlas-style review loop in which radiologists inspect the most uncertain volumes. We then count how many flagged volumes are false-positive tumor alarms with and without our training, to test whether report-derived negatives make the active-learning approach workable for tumors.