Abstract
In medical data mining, diagnostic accuracy is the most critical metric. Traditional data pre-processing methods, such as Feature Selection (FS) and Continuous Feature Discretization (CFD), often employ univariate approaches that evaluate attributes in isolation. This paper introduces a novel framework for improving classification accuracy by uncovering "latent supportive relevance"—hidden inter-correlations between attributes that are individually weak but collectively powerful. We propose two multivariate techniques: MIDCA for discretization and LUIFS for feature selection. Experimental results across 12 UCI datasets demonstrate that these methods significantly improve the performance of standard learning algorithms like ID3 and C4.5 by rescuing valuable information typically discarded during pre-processing.
1. Introduction
The effectiveness of machine learning in the medical domain is heavily dependent on the quality of data preparation. Two critical steps in this pipeline are discretization—converting continuous variables into categorical intervals—and feature selection—identifying the most relevant subset of variables for the model. However, most common pre-processing methods are univariate, meaning they ignore the interdependent relationships between features.
In clinical settings, a single symptom may be non-specific. For instance, a patient's age or weight might not be predictive of a specific condition on its own. However, when combined, they define a composite metric like Body Mass Index (BMI) or age-adjusted hypertension risk. If a pre-processing algorithm discards "age" because it lacks individual correlation with the target, the model loses the ability to form these crucial compound conditions. This paper addresses this gap by proposing methods to identify and retain features with "latent supportive relevance."
2. Proposed Methodology
The authors introduce two primary algorithms designed to capture feature interactions before the training phase.
2.1 MIDCA: Multivariate Interdependent Discretization
Standard discretization methods often use global bins or simple entropy-based splits on a single variable. MIDCA (Multivariate Interdependent Discretization for Continuous Attribute) instead seeks an "interdependent attribute" to guide the discretization process. By pairing a continuous variable with another relevant attribute (identified via normalized Information Gain and Relief measures), MIDCA determines cutting points that reflect the contextual relationship between the two, ensuring that the resulting discrete intervals are more diagnostically meaningful.
2.2 LUIFS: Latent Utility of Irrelevant Feature Selection
LUIFS is a two-phase filter designed to prevent the premature removal of features that possess latent utility.
- Attribute Sorting Phase: Features are ranked using Information Gain. Those falling below a calculated mean threshold are initially labeled "irrelevant."
- Latent Utility Searching Phase: Rather than discarding these irrelevant features, LUIFS evaluates them in pairs with other features using a combinatorial Relief algorithm. If an "irrelevant" feature significantly boosts the predictive power of another feature, it is "rescued" and included in the final optimal feature set.
3. Experimental Evaluation
The proposed methods were evaluated using 12 datasets from the UCI Machine Learning Repository, including medical datasets such as Cleve (heart disease), Hepatitis, Hypothyroid, and Diabetes. The performance was measured using the classification error rates of ID3 and C4.5 decision tree algorithms.
4. Results and Discussion
The experimental results confirm that capturing latent relevance leads to superior model performance.
- Discretization Gains: Using MIDCA reduced the average classification error for the ID3 algorithm from 15.27% (using standard univariate discretization) to 13.36%. For the C4.5 algorithm, the error rate dropped from 16.18% to 13.49%, representing a relative improvement of approximately 16.6%.
- Feature Selection Gains: LUIFS consistently outperformed both "No Feature Selection" (NoFS) and standard "Attribute Ranking Feature Selection" (ARFS). For ID3, LUIFS improved accuracy by approximately 11% over both benchmarks. In the case of C4.5, which includes internal pruning mechanisms, LUIFS still provided an 8.5% improvement over NoFS.
- Feature Rescue: The LUIFS algorithm successfully identified and retained between 1 and 8 "latent" features per dataset—variables that standard univariate filters would have discarded but which proved essential for accurate classification.
5. Conclusion and Future Work
Uncovering latent supportive relevance is a key factor in maximizing the performance of learning algorithms in data-sensitive fields like medicine. By shifting from univariate to multivariate pre-processing, MIDCA and LUIFS preserve hidden valuable information that traditional methods overlook.
While these methods introduce higher computational complexity—$O(N^2)$ due to the pairwise evaluation of features—this is often a justifiable trade-off for the gains in diagnostic accuracy. Future research will focus on optimizing these algorithms for high-dimensional data and testing their efficacy with other learning models, such as Naive-Bayes and Artificial Neural Networks.
Summary updated