Most public lung-cancer datasets do not say whether a nodule was cancer. They say what radiologists suspected. In LIDC-IDRI, the standard lung CT benchmark, up to four radiologists scored each nodule on a 1-to-5 Likert scale, an ordered rating from "highly unlikely" to "highly suspicious" for malignancy. Radiologists score defensively because a missed cancer costs far more than a false alarm. Re-thinking and Re-labeling LIDC-IDRI showed that a classifier trained on these consensus scores reaches sensitivity 0.97 but specificity only 0.23. When the same authors checked against pathology, more than half of the nodules scored 4 turned out to be benign, and nodules scored 3 were mostly benign. Their fix treats the bias as one global distortion and corrects it by borrowing labels from a separate biopsy-confirmed dataset.
We propose that the bias is not uniform but depends on where the nodule sits. Radiologists should over-call most where a nodule's surroundings look like cancer: close to vessels, touching pleura, or entangled with airways. Relationship between pulmonary nodule malignancy and surrounding pleurae, airways and vessels found these same contexts statistically associated with malignancy. If that is true, the rating scale means different things in different anatomical contexts. The debiasing threshold should then be estimated separately for each context rather than once for the whole dataset. To find those contexts without fixing their number in advance, and to set aside nodules that fit no context, we group nodules using evolutionarily stable strategies. An ESS is a population state that no small group of "mutant" alternatives can invade. Clustering methods built on this idea keep a group only if every outside point is less compatible with it than its own members are, and they leave outliers unassigned.
Core Claim
The claim has three parts. First, the gap between radiologist Likert scores and pathology in LIDC-IDRI varies systematically with anatomical context. Second, a rater model whose score cut points are specific to each context removes more of the bias than a model with one global correction. Third, the correction can be calibrated using only LIDC-IDRI's own pathology-confirmed subset, with no external anchor dataset. The ESS grouping has a narrower role. It supplies the context strata and an abstention rule for nodules that belong to no stable group, which is the principled treatment of the ambiguous midpoint score of 3. Each part can fail on its own, and each has an ablation that tests it.
Research Program
The first step computes an anatomical context description for every LIDC-IDRI nodule. We reuse the protocol of the pleura/airway/vessel study: automated segmentation of lung fields, airways and vessels, then a spherical region around each nodule centre with radius equal to the nodule diameter and at least 6 mm. From this region we compute distance to the nearest artery and vein, airway branch count and pleural contact. To this we add a learned appearance embedding of the nodule, trained with self-supervision so that no Likert score enters it. Keeping ratings out of the grouping is deliberate: similarity built from the scores would copy their bias into the groups.
The second step groups nodules. We build a similarity hypergraph over nodules, meaning a graph whose edges join more than two nodules at once. Each edge's weight is computed from the context and appearance descriptions of the nodules it joins. We then extract ESS clusters with the Baum-Eagon dynamics of HoCluGame, a hypergraph clustering method in which each cluster is an ESS. HoCluGame kept an F-measure near 1.0 with up to 40 outliers, while competing methods fell to 0.2–0.6, and it needs no preset cluster count. It has never been run on medical data. Its sequential "peeling-off", which removes one found cluster before searching for the next, is acceptable here because context strata do not need to overlap. We use the trajectory cut-off of Chehreghani's replicator-dynamics clustering to keep runtime manageable on roughly 1,500 nodules. Nodules left outside every ESS are flagged as unstable context.
The third step fits the rater model. Each nodule $n$ has a latent suspicion $z_n$. Rater $r$ reports Likert score $k$ when $z_n$ falls between cut points $\tau_{r,k} + \delta_{c(n),k}$, where $\delta_{c,k}$ is a shift shared by all raters within context cluster $c$. The rater-specific part $\tau_{r,k}$ can be estimated from disagreement between raters. The shared part $\delta_{c,k}$ cannot, because a bias that every rater shares leaves no disagreement to learn from. Only pathology can pin it down. We therefore calibrate the $\delta_{c,k}$, and a link from $z_n$ to malignancy, on the pathology-confirmed LIDC subset reported in the file (113 patients). Cluster shifts are shrunk toward a global shift so that small clusters do not overfit. The debiased labels, together with an abstain label for unstable-context nodules, then train a standard 3D classifier.
We deliberately do not build a replicator-dynamics model of radiologists "evolving" defensive thresholds. LIDC-IDRI records one reading per rater and no temporal learning. Such a model could not be tested on this data.
Baselines
The naive reference is the usual consensus label: mean Likert score, benign if below 3 and malignant if above 3, with nodules scored exactly 3 excluded, as in the pleura/airway/vessel study. A classifier trained on consensus-style labels is the one Zhang et al. report at sensitivity 0.97 and specificity 0.23. We re-run it rather than copy that number.
The strongest existing method is Zhang et al.'s metric-learning label propagation. It trains a Siamese 3D ResNet-18 on the 180 biopsy-proven SCH-LND nodules and relabels each LIDC nodule by a weighted vote of its top 20% most similar anchors. After relabeling it reports sensitivity 0.67 and specificity 0.66, trained on relabeled LIDC-IDRI. It is the strongest competitor because it targets exactly this bias and already doubles specificity. We run it with the same downstream classifier and the same tuning budget. The two methods rely on different anchor sets, so we compare them in both directions. In one direction, Zhang's method uses SCH-LND as anchors and ours uses the LIDC pathology subset, and both are tested on held-out LIDC pathology folds. In the other, both are tested on SCH-LND, which ours never sees.
The strongest statistical baseline is an ordinal Dawid-Skene-style rater model. It has per-rater cut points plus one global calibrated shift, in the spirit of the generalized worker-reliability and task-difficulty models analysed in Reliable Crowdsourcing under the Generalized Dawid-Skene Model. It gets the same pathology subset for calibration. It isolates our first key ingredient, context specificity.
A second ablation replaces the discrete ESS clusters with cut-point shifts that vary smoothly with the context features, fitted as covariates. A third replaces ESS grouping with k-means, the number of clusters chosen by BIC, and no abstention. Beating the Zhang baseline means higher balanced accuracy at matched sensitivity on held-out pathology. Beating the ablations means the same comparison against each alternative, with patient-level bootstrap intervals that exclude zero.
Why not just…?
Why not just average the radiologists' scores, or take a majority vote? Averaging cannot cancel a bias that every rater shares. The 0.23 specificity reported by Zhang et al. is what that bias looks like after aggregation. Averaging also throws away the midpoint nodules, which pathology shows are mostly benign. This baseline is included, and we expect it to lose clearly.
Why not just fit an ordinal rater model with one global correction? This is the real threat. If the bias turns out to be uniform across anatomy, this simpler model will match ours, and our central claim is false. The comparison between the global-shift model and the context-shift model settles it directly. We also run a descriptive check first: plot the gap between mean Likert score and pathology against vessel distance and pleural contact. If the gap is flat, we report a negative result instead of the method.
Why not just use k-means or smooth covariates instead of ESS clustering? They may well do equally well at debiasing. ESS clustering earns its place only through abstention: the ability to decline nodules in unstable contexts, supported by HoCluGame's outlier robustness in its original evaluation. We therefore judge it on selective-prediction curves, plotting accuracy against the fraction of nodules the model answers, and on whether abstained nodules are enriched for score-3 cases and pathology errors. If k-means with a confidence threshold matches it, we will say so and recommend the simpler grouping.
Why not just use Zhang's biopsy-anchor propagation? It works, but it needs an external histology set and gives no account of where the bias lives. Our method uses the target dataset's own pathology subset and yields per-context bias estimates. The cross-anchor comparison in both directions shows whether that costs accuracy.
Proposed Experiments
The decisive first experiment is descriptive. On the LIDC-IDRI pathology subset, split into patient-level folds, we regress each rater's score-minus-pathology gap on context features, controlling for nodule size. A significant context interaction licenses the rest of the program. The main experiment trains the same 3D classifier on the labels produced by each method and reports sensitivity, specificity, AUC and balanced accuracy. It is evaluated on held-out LIDC pathology folds and, as an external test, on SCH-LND. For ESS abstention we report risk-coverage curves and the score-3 enrichment described above.
One confound needs its own check. The pleura/airway/vessel study itself warns that its airway signal may reflect how nodules were selected, so we repeat the analysis with airway features removed. This tests whether any context effect depends on that possibly spurious signal. There is no reported result on Lacuna for AUC on the LIDC-IDRI pathology subset for any of these label-correction methods, so the AUC comparison is new measurement rather than a match against a published figure.
References
- Re-thinking and Re-labeling LIDC-IDRI for Robust Pulmonary Cancer Prediction
- Relationship between pulmonary nodule malignancy and surrounding pleurae, airways and vessels: a quantitative study using the public LIDC-IDRI dataset
- A Game-Theoretic Approach to Hypergraph Clustering
- Adaptive trajectory analysis of replicator dynamics for data clustering
- Reliable Crowdsourcing under the Generalized Dawid-Skene Model