Handling missing values in support vector machine classifiers

Kristiaan Pelckmans, Jos De Brabanter, Johan A. K. Suykens, Bart De MoorNeural Networks 2005 · DOI

TL;DR

This paper introduces a novel approach to training Support Vector Machines (SVMs) and Least Squares SVMs (LS-SVMs) when input data contains missing values. Instead of discarding incomplete data or using simple imputation, they propose a modified risk function that explicitly accounts for the uncertainty introduced by missing inputs, leading to improved classification performance.

What Problem Does This Paper Address?

In real-world machine learning applications, datasets often suffer from missing values. Traditional methods for handling this issue include:

  • Complete Case Analysis: Discarding any data point with even a single missing value. This can lead to a significant loss of valuable information and biased results, especially if the missingness is not entirely random.
  • Imputation: Filling in missing values with estimates (e.g., mean, median). While this retains more data, simple imputation can introduce bias by failing to capture complex relationships between variables.
  • Model-based Approaches: Using statistical models (like maximum likelihood) to handle missing data. These can be complex and computationally intensive.

The authors identify a gap in existing methods for non-parametric predictive settings, particularly for SVMs, which are sensitive to data quality. They aim to develop a method that can effectively leverage incomplete data without resorting to discarding it or relying on potentially biased imputation techniques.

Methods

The core of the paper's contribution lies in a new way of defining the "risk" associated with a classifier when inputs are missing. Instead of the standard empirical risk, they propose a "minimal risk" model that incorporates the uncertainty of missing inputs.

  • Modified Risk Function: They define a risk function that penalizes errors but also accounts for the fact that when an input is missing, the prediction is less certain. This is achieved by considering the expected outcome over the possible values of the missing input.
  • Non-parametric Perspective: The approach is framed from a non-parametric viewpoint, meaning it doesn't assume a specific underlying distribution for the data.
  • Additive Models and Componentwise Kernels: For multivariate data (multiple input features), they extend their approach by using additive models. This means the overall prediction is a sum of contributions from individual features. They then develop "componentwise kernel machines" where each feature's contribution is handled independently, allowing for a more nuanced treatment of missingness per feature.
  • LS-SVM Formulation: The method is efficiently implemented using the Least Squares Support Vector Machine (LS-SVM) framework, which often leads to faster training times and a more tractable optimization problem compared to standard SVMs.
  • Primal-Dual Optimization: They derive primal-dual formulations for their modified SVM and LS-SVM classifiers, which are standard techniques for efficiently solving SVM-related optimization problems.

Datasets

The paper evaluates their proposed methods on:

  • Artificial Dataset: A modified version of the Ripley dataset was used. This dataset was synthetically altered to introduce missing values, allowing for a controlled comparison of different methods.
  • Benchmark Dataset: The Hepatitis dataset from the UCI repository was used. This is a real-world dataset with a significant number of missing values, providing a more realistic test case.

Key Insights and Ideas

  • Uncertainty is Key: The most significant insight is that instead of trying to "guess" missing values, it's more effective to explicitly model the uncertainty they introduce into the prediction. This leads to a more robust and accurate model.
  • No Need to Recover Missing Values: The paper argues that the goal isn't to perfectly reconstruct missing inputs, but rather to ensure the prediction mechanism correctly accounts for their absence.
  • Componentwise Treatment for Multivariate Data: For datasets with multiple features, treating each feature's missingness independently within an additive model framework is more effective than a global approach. This allows the model to adapt to different patterns of missingness across features.
  • Efficiency through LS-SVMs: Leveraging the LS-SVM framework provides a computationally efficient way to implement these novel risk functions, making them practical for larger datasets.

Key Results

The paper demonstrates that their proposed methods generally outperform standard SVMs (which discard incomplete data) and simple imputation techniques (like median imputation) on both artificial and real-world datasets.

  • Improved Performance: On the artificial Ripley dataset, their modified SVM and LS-SVM methods showed better performance than complete case analysis and median imputation.
  • Hepatitis Dataset Success: On the Hepatitis dataset, the componentwise LS-SVM handling missing values achieved the highest classification accuracy, outperforming other methods.
  • Qualitative Insights: Figure 3 shows that their method can also provide insights into which features are most important for classification, even when some data is missing.

Limitations and Future Work

  • MCAR Assumption: The primary assumption made in this paper is that the missing values are "Missing Completely At Random" (MCAR). This means the probability of a value being missing is independent of all other variables, including the observed ones. The paper acknowledges that in many real-world scenarios, data might be "Missing At Random" (MAR) or "Missing Not At Random" (MNAR), which would require more complex modeling.
  • Additive Model Structure: The extension to multivariate data relies on the assumption of additive models. While powerful, this structure might not capture all complex interactions between features.
  • Future Work: The authors suggest that extending their framework to MAR and MNAR scenarios would be a valuable next step. Further exploration of how their method can provide feature selection and structure detection in the presence of missing data is also implied.

Summary updated

BibTeX
@article{pelckmans2005handling,
  author = {Kristiaan Pelckmans and Jos De Brabanter and Johan A. K. Suykens and Bart De Moor},
  title = {{Handling missing values in support vector machine classifiers}},
  journal = {Neural Networks},
  year = {2005},
  doi = {10.1016/j.neunet.2005.06.025},
}
Figure