Ontology-Shared Depth Context for Fine-Grained CT Lesion Labels

When a radiologist decides whether a bright spot on a CT slice is a lung nodule or a vessel seen end-on, they scroll a few slices up and down. A nodule appears and disappears; a vessel continues. Most automatic lesion-labeling systems never get to scroll. They classify one 2D slice. Systems that do look at neighboring slices give every label the same amount of depth, whether the label is "chest", which a single slice settles, or "calcified" or "lymph node versus vessel", which may need the third dimension. We propose to let the medical ontology, the tree of labels running from body part to organ to lesion type to attribute, decide how far along the scan's depth axis each label looks. Each label learns its own window of neighboring slices. That window starts from its parent's window and departs from it only when the data justify the change. Rare fine-grained labels therefore inherit a sensible window from well-populated ancestors instead of learning one from a handful of examples.

The core claim is that the need for through-plane context is organized hierarchically. Coarse labels need almost none. Some sibling families need a lot, notably those that separate tubular from focal structures and shape-defined lesion types. Sharing a depth window along the tree should improve accuracy on rare fine-grained labels compared with both single-slice classification and uniform multi-slice context, and it should cost fewer processed slices on average. The proposal is also a measurement. We will produce, for the first time, a per-level map of where in a radiology lexicon volumetric context actually matters.

What exists and what is missing

The closest work on the hierarchy side is Fine-grained lesion annotation in CT images with knowledge mined from radiology reports. It mines 145 RadLex labels from reports, propagates positive labels to ancestors (so "lung nodule" also counts as "lung" and "chest"), and trains a multi-scale VGG-16 with a noise-robust loss. It reaches an average AUC of 0.9083 on 1,872 DeepLesion test lesions. That system sees isolated 2D slices only. Its weakest labels are attributes such as "calcified" at 0.7213 AUC. Its body-part labels are already near ceiling: paraspinal reaches 0.9947. We reuse its label tree, its test set and its loss. The change is that we give it depth.

On the volumetric side, 3DCE passes neighboring slices through a shared 2D backbone and fuses the features. With 27 slices it raises DeepLesion detection sensitivity at 4 false positives per image from 80.32% to 84.37%. That establishes that depth helps lesions in general, but 3DCE applies one context width to everything. CARVE allocates visual tokens unevenly along depth, deciding by slice redundancy. It also finds that in-plane resolution often buys more than extra slices. Its allocation does not know what question is being asked. Our proposal makes allocation depend on the label being predicted, and CARVE's finding is a warning we take seriously: for many labels the best depth window may be one slice.

Three recent bridging systems show the gap clearly. CT-GRAPH pools 3D features into a three-tier anatomical graph built from TotalSegmentator masks and feeds it to a language model for report generation, gaining 7.9 F1 points over CT2Rep on CT-RATE. Its hierarchy is a fixed organ graph used for pooling. It does not encode disease relations or decide how much depth each concept sees. UniBrain splits brain MRI reports into tiers and aligns each tier with modality-level volume vectors. Its whole-volume vectors leave fine through-plane structure unconstrained. A Unified 2D Framework for DeepLesion Detection, Segmentation and Short Report Generation collapses anatomy into 11 coarse tags and works on single slices. Its authors name the move to 3D context as the needed next step.

Hierarchy-sharing as a means of transferring to sparse child nodes is supported in other settings. TrialBench moves parameters from documented parents to data-poor children, and SMI shows that ancestor-aggregated codes preserve hierarchical proximity far better than flat embeddings (Spearman 0.913 versus 0.473). We apply the same principle to a new kind of parameter: how much spatial context a concept needs.

Core mechanism

Each lesion is represented by its key slice and up to $K$ slices on each side, all encoded by a shared 2D backbone as in 3DCE.

For every label node $v$ in the RadLex tree, the model scores each slice offset $d \in \{-K,\dots,K\}$ and turns the scores into weights $a_v(d)$ with a softmax. The weighted sum of slice features feeds that label's classifier.

The scores are tied to the tree:

$$g_v(d) = g_{\mathrm{pa}(v)}(d) + \delta_v(d), \qquad \mathcal{L}_{\mathrm{tree}} = \lambda \sum_v \frac{\|\delta_v\|^2}{\log(1+n_v)}$$

Here $g_v(d)$ is the depth score of node $v$ at offset $d$, $\mathrm{pa}(v)$ is its parent, $\delta_v$ is the node's learned departure from its parent, $n_v$ is the number of positive training lesions for $v$, and $\lambda$ sets the strength of sharing.

The penalty pulls rare nodes toward their parent's window. Frequent nodes can still learn their own. The $1/\log(1+n_v)$ scaling follows the usual pattern for hierarchical shrinkage priors, in which the evidence needed to deviate grows as data shrink.

We will also add a consistency term. It penalizes child probabilities that exceed parent probabilities, which keeps predictions hierarchically valid. Report-derived training labels are noisy, so the soft bootstrapping loss from Yan et al. is kept unchanged.

A compute variant truncates each node's window at its learned 95% mass. Slices outside every active node's window are never encoded, and we report the resulting average number of slices processed.

Research Program

The first stage is diagnostic, and it is worth publishing even if the method gives no gain. For each label we train the Yan et al. classifier with depth windows of 1, 3, 9 and 27 slices, under uniform weighting. We then plot the change in AUC against tree level and against sibling family.

Our prediction is as follows. Body-part labels will gain close to zero. Lesion-type families that confuse focal and tubular structures will gain most, examples being nodule, lymph node and vessel-adjacent masses. Shape attributes will gain moderately. Intensity attributes such as "calcified" will gain little, because calcification is visible on one slice. The prediction is falsifiable. If gains are flat across levels, the premise of ontology-shared windows fails and the program stops there.

The second stage trains the shared-window model and compares it against the baselines below. The third stage checks whether the learned windows are physically meaningful. DeepLesion supplies lesion diameters and slice spacing. If the method captures something real, a node's learned window width in millimeters should correlate with the typical through-plane extent of lesions carrying that label. A null correlation would mean the windows are fitting dataset quirks.

Baselines

The naive reference point is the single-slice classifier of Yan et al., retrained in our pipeline with the same backbone, the same 145 labels, the same 1,872 manually checked test lesions and the same soft bootstrapping loss. It must reproduce close to the reported 0.9083 average AUC before any comparison counts.

The strongest competitor is uniform multi-slice fusion in the style of 3DCE attached to the same 145-label heads, with the slice count tuned from {3, 9, 27} on validation. It is the strongest competitor because it already captures most of whatever depth helps on average. The only reported number on this benchmark lineage is for detection, 84.37% sensitivity at 4 false positives per image. No reported result was found on Lacuna for multi-slice fine-grained 145-label classification on DeepLesion, so we will establish that number ourselves.

The key ablation is flat per-label depth attention: the same per-label windows with $\lambda = 0$, so no sharing across the tree. This isolates the contribution of the ontology.

A further baseline is a small 3D CNN trained on a lesion-centered crop of the volume, as an alternative to 2D backbones with fused slices.

All methods get the same training data, the same backbone initialization, the same number of tuning trials, and matched GPU hours. The 3D CNN gets a crop size chosen on validation.

We will count the proposal as beating the competitors only under one of two conditions. The first is a statistically significant AUC improvement (bootstrap over test lesions) on the bottom frequency tercile of labels, measured against both uniform fusion and flat attention, with no loss on the top tercile. The second is matched average AUC with substantially fewer slices processed per lesion. Average AUC alone is not the target, because the head labels are near ceiling.

Why not just…?

Why not just give every label 27 slices, as 3DCE does? This is the real threat. On frequent labels uniform fusion may well match us, because a well-fed classifier can learn to ignore irrelevant slices. Two things can still separate the approaches. First, on rare labels, extra irrelevant slices add input the classifier cannot learn to ignore from few examples. Second, uniform fusion pays full compute for labels that need one slice, and CARVE's observation that in-plane detail often beats slice count suggests that cost is often wasted. If the rare-tercile comparison shows no difference and the compute saving is small, we will report that uniform context suffices. The diagnostic map from stage one stands on its own either way.

Why not just learn a separate depth window per label with no hierarchy? The flat-attention ablation answers this directly. For labels with many examples we expect flat and shared windows to converge. The hierarchy should matter only where $n_v$ is small, which is the regime in which TrialBench and SMI found parent-to-child sharing useful. If flat attention matches us on rare labels, the ontology adds nothing here and we will say so.

Why not just run a 3D network on a cropped subvolume? A 3D crop sees depth natively. However, it gives up the 2D ImageNet-style pretraining that DeepLesion-scale label sets rely on. It also does not tell us which concepts need depth, and that question is half of this proposal. We include it as a baseline at matched compute. If it wins outright, the right follow-up is to apply ontology-shared receptive fields inside a 3D network, not to abandon the hierarchy question.

Decisive checks

Three outcomes would settle the program. The stage-one map must show context gains that vary with tree level and sibling family; flat gains end the premise. The shared-window model must beat both uniform fusion and flat per-label attention on the rarest third of labels at matched compute. Learned window widths must track measured lesion z-extent across label families. If all three hold, the result supports a general design rule: an ontology can tell a volumetric model where to spend its depth, which directly addresses the volumetric context loss of slice-based pipelines. As a secondary test of generality, we will move the learned per-level windows onto the 2D search-and-segment pipeline of Cheng et al. and check whether lesion-type report tokens become more accurate. That pipeline's own stated next step is to add cross-slice context.