Abstract
What dataset features affect machine learning (ML) performance has primarily been unknown in the current literature. This study examines the impact of tabular datasets' different meta-level and statistical features on the performance of various ML algorithms. We analyzed 200 open-access tabular datasets and developed multiple regression models to explore the impact of dataset features on ML performance. We found that kurtosis has a statistically significant negative effect on the accuracy of non-tree-based ML algorithms (SVM, LR, and KNN). Conversely, tree-based algorithms (Decision Tree and Random Forest) showed remarkable robustness to these statistical variations.
Introduction
In the machine learning community, there is a wealth of literature on how to improve algorithms, but far less on why certain datasets favor specific models. While practitioners intuitively know that "data quality matters," there is a lack of formal understanding regarding which specific statistical properties—beyond size or class imbalance—actually dictate performance. This research bridges the gap by quantifying the relationship between a dataset's "meta-features" (size, attribute count, class ratio) and its "statistical features" (mean, standard deviation, skewness, and kurtosis) across a wide variety of real-world tasks.
Methods
The researchers employed a multi-step pipeline to analyze 200 open-access tabular datasets (147 from Kaggle and 53 from the UCI Machine Learning Repository). These datasets spanned diverse contexts, including healthcare, finance, and sports, all framed as binary classification problems.
Feature Quantification
For each dataset, four statistical moments were calculated: mean, standard deviation, skewness, and kurtosis.
Because datasets contain multiple attributes, two aggregation methods were used to quantify these features:
- Uniform: A simple average of the statistical features across all attributes.
- Weighted: Using Principal Component Analysis (PCA) scores to weight attributes by their relative importance to the classification task.
Model Benchmarking and Regression
Five classic algorithms were trained: Support Vector Machines (SVM), Logistic Regression (LR), K-Nearest Neighbors (KNN), Decision Trees (DT), and Random Forests (RF). Multiple linear regression was then used to treat the dataset features as independent variables and model accuracy as the dependent variable, isolating which features statistically drive performance.
Key Insights and Results
The Kurtosis Curse
The most consistent finding was that kurtosis has a statistically significant negative impact on non-tree-based models. High kurtosis implies "heavy tails" or a high frequency of outliers.
Because SVMs rely on boundary distances and KNN relies on local neighborhoods, these outliers act as noise that pulls the model away from the optimal decision boundary. Kurtosis negatively impacted the accuracy of SVM, LR, and KNN at the $p \le 0.05$ level across almost all experimental variants.
Tree-Based Immunity
Interestingly, Decision Trees and Random Forests were almost entirely unaffected by the statistical distribution of the data. Because these models use hierarchical partitioning rather than distance or linearity assumptions, they are naturally "immune" to the tail-behavior (kurtosis) and asymmetry (skewness) that plague their counterparts.
The "Size" Paradox and Class Imbalance
While "more data" is often considered better, the study found that dataset size had an inconsistent impact on SVM and LR performance. Furthermore, when extremely imbalanced datasets were removed, the impact of mean and standard deviation became clearer, showing that for balanced data, the central tendency of features significantly influences non-tree model success.
Conclusion
This study highlights that the intrinsic "shape" of data is a critical bottleneck for certain model types. For practitioners, the takeaway is clear: if a dataset exhibits high kurtosis (heavy tails), tree-based models like Random Forest should be the primary choice for optimal accuracy. Future work will explore whether modern ensemble methods like XGBoost or LightGBM inherit the robustness of basic tree models or the sensitivity of non-tree models.
Summary updated