Feature selection and classification of imbalanced datasets

Édouard Duchesnay, Arnaud Cachia, Nathalie Boddaert, Nadia Chabane, Jean-Fran�ois Mangin, Jean‐Luc Martinot, Françis Brunelle, Mônica ZilboviciusNeuroImage 2011 · DOI

TL;DR

This paper tackles the challenge of classifying imbalanced datasets, specifically using PET scans to distinguish children with Autism Spectrum Disorder (ASD) from controls. They introduce a novel multi-stage feature selection and classification algorithm that combines regional feature extraction, multivariate feature selection, and an adaptive penalized likelihood model selection, achieving high accuracy and sensitivity in this difficult classification task.

What Problem Does This Paper Address?

Classifying imbalanced datasets, where one class significantly outnumbers another, is a persistent challenge in machine learning. Standard discriminative methods tend to be biased towards the majority class, leading to poor performance on the minority class. This is particularly problematic in medical imaging, where rare conditions or specific patient subgroups might be of great interest. The authors focus on applying this to neuroimaging data, specifically PET scans of children with ASD, a disorder characterized by complex and distributed brain abnormalities, making it difficult to identify a consistent neuroimaging profile. Traditional methods often struggle with the high dimensionality of brain imaging data, leading to issues like the "curse of dimensionality" and overfitting.

Methods

The authors propose a multi-stage approach:

  • Regional Feature Extraction: Instead of analyzing individual voxels, they first group statistically significant voxels into regions. This reduces dimensionality and leverages spatial information. This step involves a univariate feature selection based on p-values from F-tests, followed by thresholding and clustering.
  • Multivariate Feature Subset Ranking: They then employ a sequential floating forward selection (SFFS) algorithm to rank these regional features. This is a wrapper method that uses a multivariate F-statistic to evaluate feature combinations.
  • Adaptive Penalized Likelihood Model Selection: A key innovation is their model selection procedure. Instead of relying on fixed criteria like BIC or AIC, they develop an adaptive penalized likelihood approach. This method calibrates a penalty term based on randomized data to better account for overfitting introduced by the preceding feature selection steps.
  • Generative Classification: Finally, they use a linear discriminant analysis (LDA) classifier. LDA is a generative model that models the class-conditional densities, providing a formal framework to handle class imbalance by controlling prior probabilities.

Datasets

The study utilized a dataset of 45 children with idiopathic autism spectrum disorder (ASD) and 13 non-ASD low-functioning children. The ASD group consisted of 37 boys, aged 5-12 years. The non-ASD group also comprised children aged 5-15 years. The diagnosis of ASD was based on DSM IV-R criteria and confirmed by ADI-R scores.

Key Insights and Ideas

  • Regionalization is Key for High-Dimensional Data: Moving from individual voxels to regional features significantly aids in handling the high dimensionality of brain imaging data and potentially captures more robust patterns.
  • Adaptive Penalization Outperforms Fixed Criteria: The adaptive penalized likelihood model selection is crucial for preventing overfitting, especially after multiple feature selection steps, and demonstrates superior performance compared to standard methods like BIC and AIC.
  • Generative Models Excel with Imbalance: The use of a generative model (LDA) proved more effective than discriminative models (SVM, Lasso) in handling the imbalanced dataset, suggesting that explicitly modeling class distributions is beneficial.
  • A Parsimonious Pattern for ASD: The algorithm identified a surprisingly small set of two regions (right Superior Temporal Sulcus and left postcentral area) that were highly predictive of ASD status, suggesting that specific functional abnormalities, rather than diffuse changes, might be key biomarkers.

Key Results

The proposed multi-stage algorithm achieved:

  • High Accuracy: 88% correct classification (51 out of 58 subjects) using leave-one-out cross-validation (LOO-CV).
  • Excellent Sensitivity: 91% sensitivity (41 out of 45 ASD subjects correctly identified).
  • Good Specificity: 77% specificity (10 out of 13 non-ASD subjects correctly identified).
  • Significant Performance: These rates were statistically significant, even when accounting for the imbalanced class sizes and random chance.
  • Superiority over Alternatives: The proposed strategy significantly outperformed 14 alternative feature selection and classification strategies in terms of accuracy, sensitivity, and specificity, particularly those that combined regional feature extraction, multivariate feature selection, and adaptive model selection.

Limitations and Future Work

  • Heterogeneity of ASD: The authors acknowledge the significant heterogeneity in intellectual efficiency within ASD. They specifically studied low-functioning children, which is a large subgroup, but suggest that future work should investigate a more representative group of patients, including those with high-functioning autism and normal intelligence.
  • Generalizability: While the results were validated through LOO-CV and random permutations, replication on independent datasets is crucial to confirm generalizability.
  • Potential for More Sophisticated Feature Extraction: The regional feature extraction method used was a simple supervised procedure. The authors suggest that more sophisticated methods, like those proposed by Fan et al. (2007), could potentially replace it.
  • Multiple Etiologies and Patterns: Given the multiple potential etiologies of ASD, the authors propose that future research should explore the possibility of multiple distinct brain patterns that may exist across the autistic spectrum.

Summary updated

BibTeX
@inproceedings{duchesnay2011feature,
  author = {Édouard Duchesnay and Arnaud Cachia and Nathalie Boddaert and Nadia Chabane and Jean-Fran�ois Mangin and Jean‐Luc Martinot and Françis Brunelle and Mônica Zilbovicius},
  title = {{Feature selection and classification of imbalanced datasets}},
  booktitle = {NeuroImage},
  year = {2011},
  doi = {10.1016/j.neuroimage.2011.05.011},
}
Figure