Dataset meta-level and statistical features affect machine learning performance

Shahadat Uddin, Haohui LuScientific 2024 · DOI

Abstract

What dataset features affect machine learning (ML) performance has primarily been unknown in the current literature. This study examines the impact of tabular datasets' different meta-level and statistical features on the performance of various ML algorithms. We analyzed 200 open-access tabular datasets and developed multiple regression models to explore the impact of dataset features on ML performance. We found that kurtosis has a statistically significant negative effect on the accuracy of non-tree-based ML algorithms (SVM, LR, and KNN). Conversely, tree-based algorithms (Decision Tree and Random Forest) showed remarkable robustness to these statistical variations.

Introduction

In the machine learning community, there is a wealth of literature on how to improve algorithms, but far less on why certain datasets favor specific models. While practitioners intuitively know that "data quality matters," there is a lack of formal understanding regarding which specific statistical properties—beyond size or class imbalance—actually dictate performance. This research bridges the gap by quantifying the relationship between a dataset's "meta-features" (size, attribute count, class ratio) and its "statistical features" (mean, standard deviation, skewness, and kurtosis) across a wide variety of real-world tasks.

Methods

The researchers employed a multi-step pipeline to analyze 200 open-access tabular datasets (147 from Kaggle and 53 from the UCI Machine Learning Repository). These datasets spanned diverse contexts, including healthcare, finance, and sports, all framed as binary classification problems.

Feature Quantification

For each dataset, four statistical moments were calculated: mean, standard deviation, skewness, and kurtosis.

Example of a negatively skewed distribution

Because datasets contain multiple attributes, two aggregation methods were used to quantify these features:

  1. Uniform: A simple average of the statistical features across all attributes.
  2. Weighted: Using Principal Component Analysis (PCA) scores to weight attributes by their relative importance to the classification task.

Model Benchmarking and Regression

Five classic algorithms were trained: Support Vector Machines (SVM), Logistic Regression (LR), K-Nearest Neighbors (KNN), Decision Trees (DT), and Random Forests (RF). Multiple linear regression was then used to treat the dataset features as independent variables and model accuracy as the dependent variable, isolating which features statistically drive performance.

Key Insights and Results

The Kurtosis Curse

The most consistent finding was that kurtosis has a statistically significant negative impact on non-tree-based models. High kurtosis implies "heavy tails" or a high frequency of outliers.

Visualizing different types of Kurtosis

Because SVMs rely on boundary distances and KNN relies on local neighborhoods, these outliers act as noise that pulls the model away from the optimal decision boundary. Kurtosis negatively impacted the accuracy of SVM, LR, and KNN at the $p \le 0.05$ level across almost all experimental variants.

Tree-Based Immunity

Interestingly, Decision Trees and Random Forests were almost entirely unaffected by the statistical distribution of the data. Because these models use hierarchical partitioning rather than distance or linearity assumptions, they are naturally "immune" to the tail-behavior (kurtosis) and asymmetry (skewness) that plague their counterparts.

The "Size" Paradox and Class Imbalance

While "more data" is often considered better, the study found that dataset size had an inconsistent impact on SVM and LR performance. Furthermore, when extremely imbalanced datasets were removed, the impact of mean and standard deviation became clearer, showing that for balanced data, the central tendency of features significantly influences non-tree model success.

Conclusion

This study highlights that the intrinsic "shape" of data is a critical bottleneck for certain model types. For practitioners, the takeaway is clear: if a dataset exhibits high kurtosis (heavy tails), tree-based models like Random Forest should be the primary choice for optimal accuracy. Future work will explore whether modern ensemble methods like XGBoost or LightGBM inherit the robustness of basic tree models or the sensitivity of non-tree models.

Summary updated

Abstract

What dataset features affect machine learning (ML) performance has primarily been unknown in the current literature. This study examines the impact of tabular datasets' different meta-level and statistical features on the performance of various ML algorithms. The three meta-level features this study considered are the dataset size, the number of attributes and the ratio between the positive (class 1) and negative (class 0) class instances. It considered four statistical features for each dataset: mean, standard deviation, skewness and kurtosis. After applying the required scaling, this study averaged (uniform and weighted) each dataset's different attributes to quantify its four statistical features. We analysed 200 open-access tabular datasets from the Kaggle (147) and UCI Machine Learning Repository (53) and developed ML classification models (through classification implementation and hyperparameter tuning) for each dataset. Then, this study developed multiple regression models to explore the impact of dataset features on ML performance. We found that kurtosis has a statistically significant negative effect on the accuracy of the three non-tree-based ML algorithms of the Support vector machine (SVM), Logistic regression (LR) and K-nearest neighbour (KNN) for their classical implementation with both uniform and weighted aggregations. This study observed similar findings in most cases for ML implementations through hyperparameter tuning, except for SVM with weighted aggregation. Meta-level and statistical features barely show any statistically significant impact on the accuracy of the two tree-based ML algorithms (Decision tree and Random forest), except for implementation through hyperparameter tuning for the weighted aggregation. When we excluded some datasets based on the imbalanced statistics and a significantly higher contribution of one attribute compared to others to the classification performance, we found a significant effect of the meta-level ratio feature and statistical mean and standard deviation features on SVM, LR and KNN accuracy in many cases. Our findings open a new research direction in understanding how dataset characteristics affect ML performance and will help researchers select appropriate ML algorithms for a possible optimal accuracy outcome.
BibTeX
@inproceedings{uddin2024dataset,
  author = {Shahadat Uddin and Haohui Lu},
  title = {{Dataset meta-level and statistical features affect machine learning performance}},
  booktitle = {Scientific},
  year = {2024},
  doi = {10.1038/s41598-024-51825-x},
}
Figure