Abstract
Support vector machines (SVMs), powerful learning methods, have been popular among machine learning researches due to their strong performance on both classification and regression problems. However, traditional SVM making use of Hinge Loss cannot deal with class imbalance problems, because it applies the same weight of loss to each class. Recently, Focal Loss has been widely used for deep learning to address the imbalanced datasets. The significant effectiveness of Focal loss attracts the attention in many fields, such as object detection, semantic segmentation. Inspired by Focal loss, we reconstructed Hinge Loss with the scaling factor of Focal loss, called FH Loss, which not only deals with the class imbalance problems but also preserve the distinctive property of Hinge loss. Owing to the difficulty of the trade-off between positive and negative accuracy in imbalanced classification, FH loss pays more attention on minority class and misclassified instances to improve the accuracy of each class, further to reduce the influence of imbalance. In addition, due to the difficulty of solving SVM with FH loss, we propose an improved model with modified FH loss, called Adaptive FH-SVM. The algorithm solves the optimization problem iteratively and adaptively updates the FH loss of each instance. Experimental results on 31 binary imbalanced datasets demonstrate the effectiveness of our proposed method.
Introduction
In many real-world applications, such as rare disease detection, fraud identification, or infrastructure monitoring, the data is inherently imbalanced. The "minority" class often contains the most critical information, yet standard machine learning algorithms tend to favor the "majority" class to maximize overall accuracy. Standard Support Vector Machines (SVMs) utilize a Hinge Loss function that treats every misclassification with equal weight. In skewed datasets, this leads to models that may achieve high nominal accuracy while failing to identify minority instances entirely.
Methodology: From Hinge Loss to FH Loss
To address this, the researchers introduced FH Loss (Focal Hinge Loss). This novel loss function bridges the gap between traditional SVMs and modern "Focal Loss" techniques used in deep learning. FH Loss modifies the standard Hinge Loss by incorporating two key elements:
- Balancing Factor ($\tilde{\tau}$): This parameter explicitly assigns higher weights to the minority class, ensuring the model does not ignore rare events.
- Modulating Factor ($(1 - \tilde{p})^\gamma$): This "focal" component down-weights "easy" examples that the model already classifies correctly with high confidence. It forces the optimization process to focus on "hard" examples—those that are misclassified or lie near the decision boundary.
The Adaptive FH-SVM Algorithm
Because the modulating factor depends on the model's current predictions, the resulting optimization problem is non-linear. The authors developed the Adaptive FH-SVM algorithm to solve this iteratively. The model is trained, its outputs are used to recalculate "focal" weights for each instance, and the model is then retrained. This iterative approach allows the use of standard, efficient Quadratic Programming (QP) solvers while achieving the benefits of a complex, adaptive loss function.
Experimental Evaluation
The proposed method was tested on 31 binary imbalanced datasets from the KEEL repository, covering domains like biology and social sciences. The datasets featured Imbalance Ratios (IR) ranging from 1.82 to 41.40. Performance was measured using the F-measure (the harmonic mean of precision and recall) and the G-mean (the geometric mean of class-specific accuracies), which are more robust than simple accuracy for imbalanced data.
Key Results
The Adaptive FH-SVM demonstrated superior performance compared to five state-of-the-art methods, including SMOTE-SVM and Weighted SVM:
- Top Ranking: Across the 31 benchmark datasets, Adaptive FH-SVM achieved the best average rank for both F-measure and G-mean.
- Robustness to Extreme Imbalance: The model showed significant improvements in cases of extreme skew. For instance, on the Pageblocks13vs4 dataset, it outperformed ensemble methods by over 20% in F-measure.
- Statistical Significance: Friedman tests confirmed that the performance gains were statistically significant across the entire test suite.
Conclusion and Future Work
Adaptive FH-SVM effectively addresses class imbalance by focusing on hard-to-classify instances and minority samples while preserving the robust "maximum margin" property of SVMs. While the iterative nature of the algorithm increases computational cost compared to a single-pass SVM, its performance on small-to-medium imbalanced datasets is state-of-the-art. Future research will focus on developing faster solvers for large-scale data and extending the FH Loss framework to multi-class problems.
Summary updated