Imbalanced Learning Based on Logistic Discrimination

Huaping Guo, Weimei Zhi, Hongbing Liu, Mingliang XuComput. Intell. Neurosci 2016 · OpenReview

TL;DR

This paper introduces ILLD, a novel approach to imbalanced learning that enhances logistic discrimination by optimizing a custom cost function, APM. This method aims to improve the classification of rare (minority) classes without sacrificing overall accuracy, a common challenge in real-world datasets.

What Problem Does This Paper Address?

Many real-world datasets suffer from class imbalance, meaning one class has significantly more examples than another. This is particularly problematic in domains like medical diagnosis (e.g., rare disease detection) where correctly identifying the minority class is crucial. Traditional classification algorithms, designed for balanced datasets, tend to perform poorly on imbalanced data, often overlooking the minority class. Existing solutions address this through data preprocessing, algorithm modification, or post-processing of predictions. However, the authors note that logistic discrimination, a powerful and interpretable model, has not been extensively explored for imbalanced learning.

Methods

The core of this paper is the proposal of a new cost function called APM (Accuracy-Precision Based Metric). Unlike the standard cross-entropy error used in logistic discrimination, APM explicitly considers the accuracies of both the positive (minority) and negative (majority) classes, along with the precision of the positive class. This is achieved by incorporating terms that reflect correctly classified instances of both classes and correctly classified positive instances.

The authors then propose an algorithm called ILLD (Imbalanced Learning Based on Logistic Discrimination) that maximizes this APM cost function. ILLD utilizes a quasi-Newton method (specifically BFGS) to iteratively update the model's parameters. The paper also mentions applying traditional sampling techniques (undersampling and oversampling) to the training data in conjunction with logistic discrimination to further improve performance.

Datasets

The paper evaluates its proposed method on 14 publicly available datasets from the UCI repository. These datasets vary in their imbalance degrees, number of examples, attributes, and classes. Some are naturally imbalanced two-class datasets, while others are derived from multi-class datasets by designating one class as positive and the rest as negative. The authors do not introduce new benchmark datasets.

Key Insights and Ideas

  • The limitations of standard accuracy: The paper highlights that accuracy alone is insufficient for evaluating models on imbalanced datasets, as a model can achieve high accuracy by simply predicting the majority class.
  • The importance of a multi-faceted cost function: The APM metric is designed to balance the performance across both minority and majority classes, as well as prioritize precision for the minority class. This is a key departure from traditional methods that might focus solely on overall accuracy or recall.
  • Logistic discrimination's potential for imbalanced learning: Despite its strengths in interpretability and generalization, logistic discrimination hasn't been a primary focus for imbalanced learning. This paper demonstrates its effectiveness when adapted with a suitable cost function.
  • The synergy of sampling and a tailored cost function: The experimental results suggest that combining sampling techniques with the ILLD algorithm can further boost performance, indicating a complementary relationship.

Key Results

The proposed ILLD method, using the APM cost function, demonstrated significant improvements in several evaluation metrics compared to traditional logistic discrimination (LD) and its sampled variants (LD-US and LD-OS).

  • Recall: ILLD significantly outperformed LD on 10 out of 14 datasets, with an average recall 0.1454 higher than LD.
  • F-measure: ILLD showed much better performance than LD-US, LD-OS, and LD, significantly outperforming them on 11, 7, and 6 datasets, respectively.
  • G-mean: ILLD significantly outperformed LD-US, LD-OS, and LD on 7, 6, and 10 datasets, respectively.
  • AUC: ILLD won on 11 (6), 12 (7), and 11 (6) datasets compared to LD-OS, LD-US, and LD, respectively.
  • Accuracy: While LD slightly outperformed ILLD on average accuracy, the paper argues this is acceptable given the focus on imbalanced learning where accuracy is a less reliable metric. ILLD still showed significantly better performance than LD-US and LD-OS on most datasets.

Limitations and Future Work

The paper acknowledges that while LD slightly outperforms ILLD in average accuracy, this is expected given the focus on imbalanced learning metrics. The authors suggest that their proposed cost function APM is appropriate for imbalanced problems and that ILLD can improve logistic discrimination performance on the positive class while maintaining high accuracy. Future work could explore other sampling techniques or more advanced cost-sensitive learning strategies in conjunction with logistic discrimination.

Summary updated

BibTeX
@inproceedings{guo2016imbalanced,
  author = {Huaping Guo and Weimei Zhi and Hongbing Liu and Mingliang Xu},
  title = {{Imbalanced Learning Based on Logistic Discrimination}},
  booktitle = {Comput. Intell. Neurosci},
  year = {2016},
  url = {https://openreview.net/forum?id=QCUL6yVhg9},
}
Figure