Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction

Botong Wu, Zhen Zhou, Jianwei Wang, Yizhou Wang

Abstract

In the literature of lung nodule classification, many studies adopt Convolutional Neural Networks (CNN) to directly predict the malignancy of lung nodules using original thoracic Computed Tomography (CT) and nodule location. However, these studies often lack interpretability regarding the internal decision-making process of the network. In this paper, we propose an interpretable and multi-task learning CNN called PN-SAMP (Pulmonary Nodule Segmentation Attributes and Malignancy Prediction). PN-SAMP not only accurately predicts malignancy but also provides semantic high-level attributes and precise nodule segmentation. By combining these tasks, the model improves performance across each individual objective. Additionally, PN-SAMP incorporates multiple window widths and window centers (WW/WC) to enrich the input information, mimicking clinical radiologist workflows. Evaluation on the public LIDC-IDRI dataset demonstrates significant improvements in malignancy classification and promising results in segmentation and attribute learning compared to state-of-the-art methods.

Introduction

Lung cancer remains a leading cause of cancer-related mortality globally, making early detection via Computed Tomography (CT) critical for patient survival. While deep learning has advanced Computer-Aided Diagnosis (CAD) systems, many existing models function as "black boxes," providing malignancy scores without clinical justification. This lack of transparency—failing to explain if a score is based on margin spiculation, internal density, or size—hinders clinical trust and adoption. Furthermore, standard models often overlook the multi-spectral nature of CT data, whereas radiologists routinely adjust window settings to evaluate different tissue densities. PN-SAMP addresses these issues by integrating segmentation and semantic attribute identification into the malignancy prediction pipeline, providing a "why" behind its decisions.

Methodology

The PN-SAMP framework is an end-to-end multi-task 3D CNN designed to provide both high accuracy and clinical interpretability.

Multi-WW/WC Input

To capture the diverse information available in CT scans, the model uses a multi-channel input representing different window widths and window centers (WW/WC). This allows the network to process information from both "lung windows" and "mediastinal windows" simultaneously, providing a richer feature set than single-channel grayscale volumes.

Architecture and Task Synergy

The core of PN-SAMP is a modified 3D U-Net backbone optimized for efficiency. With only 1.5M parameters—significantly fewer than the standard 16M parameter 3D U-Net—the model reduces the risk of overfitting while maintaining high performance.

The PN-SAMP architecture integrates segmentation, attribute learning, and malignancy prediction through interconnected sub-networks.

The architecture features interconnected sub-networks where the attribute and malignancy branches are fed by both low-level encoder features and high-level semantic features from the segmentation mask. This design facilitates task synergy: for example, learning the visual characteristics of a "spiculated margin" improves both the segmentation accuracy and the malignancy prediction.

Experimental Results

The model was evaluated on the LIDC-IDRI dataset, which includes 1,010 patients and 2,660 nodules. The analysis focused on 1,404 nodules where radiologist consensus was clear, excluding ambiguous cases.

Performance and Interpretability

PN-SAMP achieved a state-of-the-art 97.58% accuracy in malignancy prediction, a substantial improvement over previous models like MCNN (86.84%) and CNN-MTL (91.26%). The segmentation component reached a 73.89% Dice Coefficient, outperforming standard 3D U-Net implementations.

Example of a predicted nodule segmentation mask generated by the PN-SAMP model.

Beyond raw performance, the model successfully predicted nine semantic attributes (such as sphericity, margin, and lobulation) with 89.33% accuracy. These attributes provide a clinical rationale for the malignancy score, offering radiologists actionable semantic evidence. The system is also computationally efficient, with an inference time of under 0.5 seconds per nodule patch on a Titan X GPU.

Conclusion

PN-SAMP represents a significant step toward interpretable AI in oncology. By mimicking radiologist behavior through multi-window inputs and leveraging the synergy of multi-task learning, the model achieves superior accuracy while providing actionable semantic evidence. Future research will focus on validating the model against biopsy-proven pathology data and utilizing semi-supervised learning to harness larger, unlabeled datasets.

Summary updated

Abstract

Refer to the literature of lung nodule classification, many studies adopt Convolutional Neural Networks (CNN) to directly predict the malignancy of lung nodules with original thoracic Computed Tomography (CT) and nodule location. However, these studies cannot tell how the CNN works in terms of predicting the malignancy of the given nodule, e.g., it's hard to conclude that whether the region within the nodule or the contextual information matters according to the output of the CNN. In this paper, we propose an interpretable and multi-task learning CNN -- Joint learning for \textbf{P}ulmonary \textbf{N}odule \textbf{S}egmentation \textbf{A}ttributes and \textbf{M}alignancy \textbf{P}rediction (PN-SAMP). It is able to not only accurately predict the malignancy of lung nodules, but also provide semantic high-level attributes as well as the areas of detected nodules. Moreover, the combination of nodule segmentation, attributes and malignancy prediction is helpful to improve the performance of each single task. In addition, inspired by the fact that radiologists often change window widths and window centers to help to make decision on uncertain nodules, PN-SAMP mixes multiple WW/WC together to gain information for the raw CT input images. To verify the effectiveness of the proposed method, the evaluation is implemented on the public LIDC-IDRI dataset, which is one of the largest dataset for lung nodule malignancy prediction. Experiments indicate that the proposed PN-SAMP achieves significant improvement with respect to lung nodule classification, and promising performance on lung nodule segmentation and attribute learning, compared with the-state-of-the-art methods.
BibTeX
@inproceedings{wu2018joint,
  author = {Botong Wu and Zhen Zhou and Jianwei Wang and Yizhou Wang},
  title = {{Joint Learning for Pulmonary Nodule Segmentation, Attributes and Malignancy Prediction}},
  booktitle = {ISBI},
  year = {2018},
  url = {https://openreview.net/forum?id=ybR5qK6iXQ},
}
Figure