Abstract
Support Vector Machines (SVMs) are powerful classifiers but do not natively provide posterior class probabilities. Platt Scaling addresses this by fitting a sigmoid function to SVM decision values. However, the original implementation of this method suffered from numerical instability and logic errors. This technical note proposes a robust Newton-based optimization algorithm with backtracking line search to ensure convergence and numerical stability. The improved implementation eliminates overflow errors and has become the standard in major machine learning libraries like LIBSVM.
Introduction
Standard SVMs output a "decision value" representing the distance of a point from the separating hyperplane. To transform these into posterior class probabilities, John Platt proposed fitting a sigmoid function. However, the optimization problem required to find the best sigmoid parameters is numerically challenging. The original 1999 implementation used a simplified Levenberg-Marquardt (LM) algorithm that lacked a convergence proof and was prone to floating-point errors, such as overflow and catastrophic cancellation. Furthermore, a logic bug in the original pseudo-code prevented the target labels from being updated correctly, hindering the optimization process.
Methodology
We propose replacing the LM algorithm with Newton’s method with backtracking line search. Our approach is grounded in three key improvements:
- Convexity and Convergence: We prove that the optimization problem is convex, ensuring that a global minimum exists and can be reliably reached using second-order methods.
- Numerical Reformulation: To avoid
log(0)errors and numerical overflow, we reformulate the objective function. By using conditional logic and the identity $\log(1 + \exp(x))$, we ensure the computer encounters safe "underflow" (rounding to zero) rather than fatal "overflow" (resulting inINF). - Regularization: We add a small diagonal shift to the Hessian matrix to handle potential singularity, ensuring the algorithm always maintains a valid descent direction.
Experimental Results
The authors evaluated the improved algorithm using the UCI Sonar and Shuttle datasets. SVM decision values were generated using RBF kernels across various parameter settings to test the robustness of the optimization.
Stability and Efficiency
The improvements significantly impacted reliability. On the "Sonar" dataset, while all methods were stable, our Newton-based approach achieved convergence in fewer iterations.
In the more challenging "Shuttle" dataset, the original implementation encountered an average of 589.30 overflow errors per run. In contrast, our proposed method encountered zero. Additionally, our approach consistently found lower (better) objective values ($F(z)$) for the maximum likelihood problem, indicating superior accuracy in parameter estimation.
Conclusion
This technical note provides a critical update to the implementation of Platt Scaling. By addressing numerical instability and logic bugs with a mathematically rigorous Newton-based framework, we ensure that SVM probability estimation is both reliable and efficient. These improvements have been integrated into LIBSVM, providing a stable foundation for the machine learning community.
Summary updated