A note on Platt's probabilistic outputs for support vector machines

Hsuan-Tien Lin, Chih-Jen Lin, Ruby C. WengMachine Learning 2007 · DOI · OpenReview

Abstract

Support Vector Machines (SVMs) are powerful classifiers but do not natively provide posterior class probabilities. Platt Scaling addresses this by fitting a sigmoid function to SVM decision values. However, the original implementation of this method suffered from numerical instability and logic errors. This technical note proposes a robust Newton-based optimization algorithm with backtracking line search to ensure convergence and numerical stability. The improved implementation eliminates overflow errors and has become the standard in major machine learning libraries like LIBSVM.

Introduction

Standard SVMs output a "decision value" representing the distance of a point from the separating hyperplane. To transform these into posterior class probabilities, John Platt proposed fitting a sigmoid function. However, the optimization problem required to find the best sigmoid parameters is numerically challenging. The original 1999 implementation used a simplified Levenberg-Marquardt (LM) algorithm that lacked a convergence proof and was prone to floating-point errors, such as overflow and catastrophic cancellation. Furthermore, a logic bug in the original pseudo-code prevented the target labels from being updated correctly, hindering the optimization process.

Methodology

We propose replacing the LM algorithm with Newton’s method with backtracking line search. Our approach is grounded in three key improvements:

  1. Convexity and Convergence: We prove that the optimization problem is convex, ensuring that a global minimum exists and can be reliably reached using second-order methods.
  2. Numerical Reformulation: To avoid log(0) errors and numerical overflow, we reformulate the objective function. By using conditional logic and the identity $\log(1 + \exp(x))$, we ensure the computer encounters safe "underflow" (rounding to zero) rather than fatal "overflow" (resulting in INF).
  3. Regularization: We add a small diagonal shift to the Hessian matrix to handle potential singularity, ensuring the algorithm always maintains a valid descent direction.

Experimental Results

The authors evaluated the improved algorithm using the UCI Sonar and Shuttle datasets. SVM decision values were generated using RBF kernels across various parameter settings to test the robustness of the optimization.

Stability and Efficiency

The improvements significantly impacted reliability. On the "Sonar" dataset, while all methods were stable, our Newton-based approach achieved convergence in fewer iterations.

Table 1: Performance comparison on the UCI Sonar dataset.

In the more challenging "Shuttle" dataset, the original implementation encountered an average of 589.30 overflow errors per run. In contrast, our proposed method encountered zero. Additionally, our approach consistently found lower (better) objective values ($F(z)$) for the maximum likelihood problem, indicating superior accuracy in parameter estimation.

Table 2: Performance comparison on the UCI Shuttle dataset, highlighting the elimination of overflow errors.

Conclusion

This technical note provides a critical update to the implementation of Platt Scaling. By addressing numerical instability and logic bugs with a mathematically rigorous Newton-based framework, we ensure that SVM probability estimation is both reliable and efficient. These improvements have been integrated into LIBSVM, providing a stable foundation for the machine learning community.

Summary updated

Abstract

Platt’s probabilistic outputs for Support Vector Machines (Platt, J. in Smola, A., et al. (eds.) Advances in large margin classifiers. Cambridge, 2000) has been popular for applications that require posterior class probabilities. In this note, we propose an improved algorithm that theoretically converges and avoids numerical difficulties. A simple and ready-to-use pseudo code is included.
BibTeX
@article{lin2007note,
  author = {Hsuan-Tien Lin and Chih-Jen Lin and Ruby C. Weng},
  title = {{A note on Platt's probabilistic outputs for support vector machines}},
  journal = {Machine Learning},
  year = {2007},
  doi = {10.1007/s10994-007-5018-6},
}
Figure