Small cancers are missed because they are small: a pancreatic tumor under 2 cm occupies a few hundred voxels in a CT volume of tens of millions. Two-stage detectors suit this setting well. The first stage proposes a short list of suspicious regions, and the second stage looks closely at each one and decides what it is. Their weak point is that a lesion the first stage never proposes cannot be recovered later. Hospitals hold an underused training signal that speaks to exactly this weakness. The radiology report written for each scan usually states how many lesions there are, roughly how big, and where they sit ("1.4 cm hypodense lesion in the pancreatic head"). It also states negative findings ("no pancreatic mass").
We propose training a two-stage tumor detector in which these report statements supervise each stage according to what each stage needs. The count, size, and sub-organ location in a report become a recall constraint on the proposal stage: at least one proposal of roughly the right size must land in the right part of the organ. Descriptive words and explicit negations become labels for the refinement stage. At test time the detector reads only the image. The report is used only during training, so detection is not limited to what radiologists already wrote down.
Core Claim
The closest existing work, Report Supervision (R-Super), uses Llama 3.1 70B to turn CT reports into lesion counts, diameters, and sub-locations. It converts those into a volume loss and a sliding-sphere "ball loss" on a single 3D segmentation network, and it reports an F1 of 80 on pancreatic tumors of 2 cm or smaller. Its losses ask a dense voxel map to match the report everywhere at once. We claim that report content splits naturally along the two-stage boundary. Coarse statements ("one lesion, about this size, in this sub-region") are enough to supervise proposal recall. Finer statements (solid versus cystic, hypodense, "cannot exclude") are enough to supervise refinement on cropped regions. A detector trained this way should reach higher small-tumor sensitivity at a matched false-positive rate than R-Super's segmentation outputs.
A secondary claim follows. Because the proposal stage only needs count, size, and location, it should tolerate a weaker text extractor than R-Super requires. R-Super reports severe degradation when its extractor shrinks to Qwen 0.6B. This matters for deployment at hospitals that cannot run a 70B model on protected notes.
The vision-only multi-stage detector in A clinically applicable AI system for detection and diagnosis of bone metastases using CT scans (BLDS) motivates the architecture. It uses a cascaded skeleton segmentation, a feature-pyramid proposal stage, a false-positive removal module, and a ResNet classifier, and it reaches 89.1% lesion-wise sensitivity at 1.40 false positives per case on non-contrast CT. It was trained on manual masks from 2,518 scans and never reads text. We keep its decomposition and replace most of its manual supervision with reports.
Research Program
The pipeline has three parts.
Organ masks. An organ segmentation model produces pancreas head, body, and tail masks. R-Super has the same prerequisite, and PanTS shows that surrounding anatomy improves tumor segmentation by 10.3% mean Dice.
Report extraction. An extractor turns each report into a small record: lesion count $K$, a diameter $d_j$ and sub-location region $R_j$ for each lesion $j$, descriptive attributes, and negation and uncertainty flags. We use Llama 3.1 70B to match R-Super, plus a small domain model in the style of RadGraph. RadGraph uses PubMedBERT with a joint entity and relation parser, and it reaches 0.94 entity F1 and 0.82 relation F1 on chest X-ray reports. We will re-annotate a few hundred abdominal CT reports to adapt it.
Detector. The detector is a 3D two-stage model. The proposal stage is a feature-pyramid network that emits box proposals with centers $c_i$, sizes $s_i$, and objectness scores $p_i$. The refinement stage classifies cropped regions and regresses their boxes.
The proposal stage is trained with a report-coverage loss:
$$ \mathcal{L}_{\text{cov}} = -\sum_{j=1}^{K} \log \max_{i \in M_j} p_i , \qquad M_j = \{\, i : c_i \in R_j,\ |s_i - d_j| \le \tau d_j \,\}. $$
Here $M_j$ is the set of proposals compatible with reported lesion $j$, and $\tau$ is a size tolerance, playing the role of R-Super's measurement margin. This is a multiple-instance objective. It rewards the best compatible proposal for each reported lesion and does not dictate which one.
For scans whose report explicitly negates a pancreatic lesion, a second term penalizes the top-$k$ proposal scores inside the organ. This gives the detector real empty scans to learn from, which addresses the closed-world failure (assuming an object is always present) discussed in Is There Really a Camouflaged Object?.
The refinement stage receives labels in three ways:
- Proposals matched to reported lesions by bipartite matching on location and size inherit that lesion's attributes.
- Proposals in negated organs are hard negatives.
- Unmatched proposals in positive scans are down-weighted negatives, not hard ones, because reports miss lesions.
A soft count term asks that the number of confident refinement-stage detections in the organ approximate $K$.
The analysis centers on the proposal-recall ceiling: the fraction of true lesions, judged against PanTS voxel labels converted to boxes, covered by any proposal. We report it by tumor size and by contrast phase. If report supervision helps, this ceiling is where it should show first.
Baselines
Naive reference. A two-stage detector of identical architecture trained only on the PanTS voxel-label training split (9,901 scans) with boxes derived from masks. This is a BLDS-style recipe without reports. It tells us what reports add over the manual masks a public user already has.
Strongest existing method: R-Super. We run R-Super with its published losses on the same report-paired CT data. Its segmentation output is converted to detections by taking connected components, scored by mean probability. Both systems use the same backbone family, organ masks, extraction outputs, training scans, GPU budget, and hyperparameter search size. Comparison is by free-response operating characteristic (sensitivity across false-positive rates), with small-tumor sensitivity at 1 false positive per case as the headline number.
The reference figure from the literature is R-Super's F1 of 80 on tumors of 2 cm or smaller (Report Supervision). We will reproduce it rather than compare against the printed number, because our data mix may differ. Beating R-Super means higher small-tumor sensitivity at the same false-positive rate on PanTS test and on the Swiss external set. For Swiss-Test there is no reported result found on Lacuna for any method, so both systems are evaluated fresh.
Pseudo-mask distillation. We train the identical two-stage detector on boxes derived from R-Super's predicted masks. This isolates our key ingredient: supervising each stage directly from report constraints, as opposed to routing reports through a segmentation model first.
Ablations. We drop the negation term, drop the coverage loss (refinement-stage labels only), and swap the 70B extractor for the small one. Each ablation asks whether its ingredient carries the gain.
Why not just…?
Why not just threshold R-Super's segmentation map into detections? This is a real threat, and the main comparison tests it directly. R-Super's ball loss already encourages sphere-shaped high-probability regions at reported locations, so its components may behave like proposals. Our bet is that a dense map trained to match volume and count has no explicit pressure toward recall on faint lesions under a false-positive budget. A proposal stage trained with a max-over-compatible-proposals objective does have that pressure. If R-Super matches us on small-tumor sensitivity at 1 false positive per case, the decomposition adds nothing and we will say so.
Why not just train a two-stage detector on manual masks, as BLDS did? Manual masks are expensive: about 30 minutes per volume, per the BLDS paper. R-Super reaches about 20 times more scans through reports than the largest mask cohorts. PanTS shows how much data scale matters, with tumor detection AUC rising from 0.819 to 0.959 when moving from PANORAMA to the larger PanTS training set. The naive baseline measures what masks alone give. A mask-plus-report variant tests whether reports still help once masks exist.
Why not just distill R-Super's pseudo-masks into a standard detector? This is the most plausible alternative and the most important ablation. Pseudo-masks inherit the segmentation model's misses on exactly the small lesions we care about. Direct coverage supervision can still reward a proposal at a reported location the segmentation model scored low. If distillation equals our method, the contribution shrinks to the architecture, not the supervision.
Why not just read the report at test time? Early detection targets lesions the radiologist did not describe. BLDS found unaided radiologists at 40.5% sensitivity on subtle bone lesions. A report-reading system cannot exceed the report, so it can serve only as a training signal.
Proposed Experiments
Data. We train on report-paired abdominal CT: AbdomenAtlas 3.0 (9,262 scans) and the Merlin cohort (over 25,000 CT-report pairs), both used by R-Super, with the PanTS training split for the mask-supervised arms. We evaluate on the PanTS test split (26,489 scans with voxel tumor labels), stratified by size (2 cm or smaller versus larger) and by contrast phase, since non-contrast scans are where report supervision helped R-Super. External evaluation uses Swiss-Test (684 pairs). The UCSF data used by R-Super is likely inaccessible, so we do not claim parity at its scale.
Decisive check 1. Does the coverage loss raise the proposal-recall ceiling for small tumors over the naive detector, the R-Super components, and the distillation baseline? If not, the core claim is false.
Decisive check 2. We inject controlled corruption into the extracted reports: wrong sub-location, wrong count, and dropped lesions, at rates from 0 to 30%. We plot small-tumor sensitivity for our method and R-Super, which is reported robust to 10%. In parallel we compare the 70B extractor with the small adapted extractor. The secondary claim holds only if the proposal stage degrades less than R-Super under both coarse noise and the smaller extractor.
Decisive check 3. We run per-lesion analysis on PanTS test cases where the report-derived record and the voxel labels disagree, for example lesions present in the labels but absent from the report. This tells us whether down-weighting unmatched proposals lets the detector find unreported tumors, or whether it learns to reproduce report omissions.
References
- Report Supervision
- A clinically applicable AI system for detection and diagnosis of bone metastases using CT scans
- PanTS: The Pancreatic Tumor Segmentation Dataset
- RadGraph: Extracting Clinical Entities and Relations from Radiology Reports
- Is There Really a Camouflaged Object? Towards Realistic Camouflaged Object Detection