Abstract
Machine learning techniques have been developed to learn from complete data. When missing values exist in a dataset, the incomplete data should be preprocessed separately by removing data points with missing values or imputation. In this paper, we propose an online approach to handle missing values while a classification model is learnt. To reach this goal, we develop a multi-objective optimization model with two objective functions for imputation and model selection. We also propose three formulations for imputation objective function. We use an evolutionary algorithm based on NSGA II to find the optimal solutions as the Pareto solutions. We investigate the reliability and robustness of the proposed model using experiments by defining several scenarios in dealing with missing values and classification. We also describe how the proposed model can contribute to medical informatics. We compare the performance of three different formulations via experimental results. The proposed model results get validated by comparing with a comparable literature.
Introduction
Most machine learning algorithms are designed for complete datasets. However, in real-world applications—particularly in medical informatics—missing values are common. Traditionally, researchers handle this by either discarding incomplete records, which wastes valuable information, or performing "offline" imputation, where missing values are estimated before training the model.
A major flaw in the offline approach is that the imputation process is disconnected from the prediction task. An imputation method might appear statistically sound but introduce noise that degrades the performance of a specific classifier. Furthermore, the standard pipeline involves four distinct steps: imputing training data, training the model, imputing test data, and finally predicting. This paper proposes an "online" framework to collapse these steps into a single, unified process.
Methods
The researchers developed a multi-objective optimization model that searches for the best parameters for both imputation and classification simultaneously.
The Optimization Engine
The model utilizes the NSGA-II (Non-dominated Sorting Genetic Algorithm II), an evolutionary algorithm designed to find a "Pareto front" — a set of optimal solutions representing the best trade-offs between competing goals. The parameters for both the imputation model and the classifier are encoded into a single chromosome for joint optimization.
Imputation and Classification
- Imputation: The framework uses Fuzzy Clustering. Instead of hard-assigning a point to one cluster, it assigns "degrees of membership." The model optimizes cluster centers ($c_{ij}$) and the "fuzziness" parameter ($v$) to fill in missing values.
- Classification: Support Vector Machines (SVM) are employed. The algorithm tunes the SVM’s regularization parameter ($C$) and kernel parameters (such as $\gamma$, degree $d$, or coefficient $r$) alongside the imputation parameters.
Objective Functions
The model optimizes two primary goals:
- Maximize Imputation Quality: Three formulations were tested: Cluster Validity (Average Silhouette Width), Correlation (maximizing the link between train and test sets), and Variance Ratio.
- Minimize Classification Error: Measured via 10-fold cross-validation on the training set.
Experimental Setup
The model was evaluated using three classic benchmarks from the UCI repository: Iris (small, numerical), Zoo (categorical), and Sonar (larger, numerical). To test robustness, the authors generated 30 scenarios for each dataset, varying the missingness ratio from 1% to 50% and using different missingness patterns (simple, medium, or complex).
Key Results and Insights
- Minimal Accuracy Loss: Even with 50% missing data, the classification error on imputed test data was only approximately 3% higher than on complete data.
- Performance vs. Baselines: The model achieved lower or equal test errors compared to existing Train Subset Selection (TSS) methods, even when those methods utilized complete datasets.
- Pareto Efficiency: The multi-objective approach provides a range of solutions, allowing researchers to choose between prioritizing imputation reliability or maximum classification accuracy.
- Robustness: Performance remained stable across different missingness types (Missing at Random vs. Uniformly Distributed), proving the framework's applicability to various real-world data failure modes.
Discussion and Conclusion
By optimizing imputation specifically for the classifier, the model ensures that filled-in values are those most helpful for accurate predictions. This is particularly relevant in fields like medical informatics, where every patient record is valuable and expensive to obtain.
While effective, the current evolutionary approach faces scalability challenges as the number of records and features increases. Future work may involve extending the framework to include feature selection and model pruning as additional objectives, as well as testing with other architectures like Random Forests or Deep Neural Networks.
Summary updated