Logistic Regression Confined by Cardinality-Constrained Sample and Feature Selection

Ehsan Adeli, Xiaorui Li, Dongjin Kwon, Yong Zhang, Kilian M. Pohl

Abstract

Many vision-based applications rely on logistic regression for embedding classification within a probabilistic context, such as recognition in images and videos or identifying disease-specific image phenotypes from neuroimages. Logistic regression, however, often performs poorly when trained on data that is noisy, has irrelevant features, or when the samples are distributed across the classes in an imbalanced setting; a common occurrence in visual recognition tasks. To deal with those issues, researchers generally rely on adhoc regularization techniques or model a subset of these issues. We instead propose a mathematically sound logistic regression model that selects a subset of (relevant) features and (informative and balanced) set of samples during the training process. The model does so by applying cardinality constraints (via ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> -`norm' sparsity) on the features and samples. ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">0</sub> defines sparsity in mathematical settings but in practice has mostly been approximated (e.g., via ℓ <sub xmlns:mml="http://www.w3.org/1998/Math/MathML" xmlns:xlink="http://www.w3.org/1999/xlink">1</sub> or its variations) for computational simplicity. We prove that a local minimum to the non-convex optimization problems induced by cardinality constraints can be computed by combining block coordinate descent with penalty decomposition. On synthetic, image recognition, and neuroimaging datasets, we show that the accuracy of the method is higher than alternative methods and classifiers commonly used in the literature.
BibTeX
@article{adeli2020logistic,
  author = {Ehsan Adeli and Xiaorui Li and Dongjin Kwon and Yong Zhang and Kilian M. Pohl},
  title = {{Logistic Regression Confined by Cardinality-Constrained Sample and Feature Selection}},
  journal = {IEEE Trans. Pattern Anal. Mach. Intell.},
  year = {2020},
  url = {https://openreview.net/forum?id=wJloK0HLAS},
}

Loading related content...

Figure