TL;DR
This paper introduces a novel hybrid framework to evaluate both the stability and effectiveness of feature ranking (FR) algorithms for breast cancer (BC) diagnosis. They found that Generalized Fisher Score (GFS) consistently ranks features that lead to superior diagnostic performance across various datasets, highlighting the importance of shape features in image analysis and specific genes in differentiating tumor types.
What Problem Does This Paper Address?
Breast cancer (BC) is a major global health concern, and accurate diagnosis is crucial for effective treatment. Computer-aided diagnosis (CAD) models rely on identifying relevant features from complex datasets to distinguish between benign and malignant tumors. However, a significant challenge lies in selecting features that are not only predictive but also stable – meaning they consistently rank high across different data subsets or analyses. Many existing feature selection methods are evaluated solely on their predictive power, neglecting stability, which can lead to a lack of user confidence and hinder real-world deployment. This paper addresses this gap by proposing a framework that rigorously assesses both the stability and diagnostic effectiveness of a wide range of FR algorithms on BC data.
Methods
The authors developed a "stability-first" hybrid framework to systematically evaluate FR algorithms. This framework involves two main stages:
- Stability Estimation: For each FR algorithm, they performed multiple iterations of random data splitting. In each iteration, the algorithm ranks features on a training set. An advanced estimator [17] was used to quantify the stability of these rankings across iterations. A key criterion for stability was set at S ≥ 0.55.
- Effectiveness Study: For algorithms identified as stable, their top-ranked features were incrementally used to train machine learning classifiers. The predictive performance (measured by AUC, accuracy, etc.) was then evaluated on a testing set. This allowed them to assess how well stable feature subsets contribute to accurate BC diagnosis.
They evaluated 23 FR algorithms from the matFR toolbox and compared their performance using standard machine learning classifiers like KNN, LDA, NB, SVM, ANN01, and ANN02.
Datasets
The study utilized four publicly available BC datasets:
- BCDR-F03: Contains mammographic (MAM) images with 17 features derived from intensity, shape, and texture analysis.
- WDBC (Wisconsin Diagnostic Breast Cancer): Features are derived from fine needle aspiration (FNA) images, totaling 30 features.
- GSE10810: A gene expression dataset with 18,382 gene profiles.
- GSE15852: Another gene expression dataset with 22,283 gene profiles.
These datasets represent different data modalities (images and gene expression) and vary in sample size and feature dimensionality, providing a comprehensive evaluation environment.
Key Insights and Ideas
- Stability is Key: The study strongly emphasizes that feature ranking stability is as crucial as predictive power for reliable BC diagnosis. Unstable features can lead to inconsistent model performance and reduced trust.
- GFS Shines: Generalized Fisher Score (GFS) emerged as a consistently stable and highly effective FR algorithm across all tested datasets.
- Feature Type Matters: The analysis revealed that for image-based datasets (BCDR-F03, WDBC), shape features (like "contrast," "circularity," "perimeter," "concave points," and "largest perimeter") are particularly important for accurate diagnosis.
- Gene Signatures: For gene expression datasets (GSE10810, GSE15852), a small subset of genes was sufficient to differentiate between benign and malignant cases, suggesting the potential for targeted gene-based diagnostics.
- Data Sufficiency Impact: The study observed that FR algorithms generally showed better stability on image datasets with more samples relative to features, compared to gene datasets with high dimensionality and fewer samples. This highlights the need for sufficient data to reliably estimate feature importance.
Key Results
- Stable Algorithms: Three algorithms (GFS, PWFP, and LNEC) demonstrated consistent stability across all four datasets.
- Superior Performance: GFS consistently led to superior diagnostic performance (higher AUC, ACC, SEN, SPE, NPV, F-measure, and MCC) compared to PWFP and LNEC, especially on the gene expression datasets where the difference was statistically significant (p-values < 10−8).
- Feature Efficiency: GFS achieved high diagnostic accuracy using a remarkably small number of features (e.g., 2-4 genes for gene datasets, 2-4 image features for image datasets), often outperforming methods that used many more features.
- Speed: GFS was found to be the fastest FR algorithm among those tested, with an average time cost of approximately 0.12 seconds per iteration on the GSE15852 dataset.
Limitations and Future Work
The authors acknowledge several limitations:
- Training Size Impact: The study used fixed training set sizes for gene datasets due to sample limitations. Future work could explore the impact of varying training sizes on stability.
- Single Estimator: While the chosen stability estimator [17] has desirable properties, exploring more estimators and conducting comprehensive evaluations is suggested.
- Classifier Simplicity: Simple classifiers were used to maintain interpretability. Future research could incorporate more advanced classifiers, including deep learning networks.
- Feature Integration: The study focused on quantitative features. Future work could explore concatenating low-, middle-, and high-level features from deep learning models.
- Gene Analysis: While GFS identified potentially important genes, a deeper retrieval and meta-analysis of these genes for understanding cancer mechanisms was beyond the scope of this study.
The authors propose future directions including exploring different data splitting strategies, developing more advanced estimators, and integrating deep learning approaches for a more comprehensive BC diagnosis framework.
Summary updated