Unveiling Statistical Significance of Online Regression over Multiple Datasets

Mohammad Abu-Shaira, Weishi Shi

Introduction

In the rapidly evolving field of machine learning, benchmarking new algorithms against existing baselines is a fundamental task. While traditional "batch" learning often relies on standard parametric tests like the t-test or ANOVA, these methods frequently fall short in the context of Online Learning. In streaming environments, data arrives continuously, models update incrementally, and assumptions of normality or equal variance are often violated by phenomena like concept drift.

This paper addresses the challenge of rigorously comparing multiple online learning algorithms across diverse datasets. By employing non-parametric statistical frameworks—specifically the Friedman and Nemenyi tests—the authors demonstrate how to move beyond simple average performance metrics to determine if one online regression model truly outperforms another with statistical certainty.

Methodology: A Two-Step Statistical Framework

To claim a model is "state-of-the-art," researchers need a robust way to compare multiple algorithms across many different datasets simultaneously. The authors propose a two-step approach:

  1. The Friedman Test: As a non-parametric alternative to the repeated-measures ANOVA, this test ranks the performance of each model on every dataset (assigning 1 for the best and $k$ for the worst). It determines whether the observed differences in average ranks across all datasets are statistically significant or merely the result of random variation.
  2. Nemenyi Post-hoc Test: If the Friedman test indicates significant differences, the Nemenyi test is applied for pairwise comparisons. It calculates a Critical Difference (CD). If the difference between the average ranks of two models exceeds this CD, the performance gap is considered statistically significant.

The study evaluates eight online linear regression models: Stochastic Gradient Descent (SGD), Mini-Batch Gradient Descent (MBGD), Least Mean Squares (LMS), Online Ridge Regression (ORR), Online Lasso Regression (OLR), Recursive Least Squares (RLS), the Passive Aggressive (PA) algorithm, and the authors' proposed Online Regression with Weighted Average (OLR-WA).

Experimental Setup

The evaluation utilizes a diverse suite of eight datasets, encompassing both synthetic and real-world scenarios to ensure a comprehensive assessment of algorithm robustness.

Table 1: Characteristics of the synthetic and real-world datasets used in the evaluation, including dimensionality and noise levels.

The synthetic datasets (DS1–DS4) vary in dimensionality from 3 to 500 features with different noise levels. Real-world datasets include medical costs (MCPD), company profits (1KC), housing data (KCHSD), and power plant metrics (CCPP). All experiments were conducted using 5-fold cross-validation with seed averaging to ensure stability.

Key Results and Statistical Analysis

The core finding of the study is that while many models may eventually converge to similar error rates, their performance during the initial convergence phase (the first 10 iterations) reveals significant differences in adaptation speed and efficiency.

Table 2: Detailed performance comparison showing MSE and rank for each algorithm across all datasets, along with average ranks.

The Friedman test yielded an $F_F$ statistic of 7.63, which significantly exceeds the critical value of 3.047. This result allows for the rejection of the null hypothesis, confirming that the performance differences between the models are statistically significant.

OLR-WA emerged as the top performer with the lowest average rank of 1.250. It was followed by the PA and RLS algorithms, both achieving an average rank of 3.875. The Nemenyi test further clarified these relationships: while OLR-WA significantly outperforms SGD, MBGD, and ORR, the gap between OLR-WA and LMS (3.125) was less than the Critical Difference (3.712 at $\alpha=0.05$), meaning OLR-WA cannot be claimed as statistically superior to LMS in this specific experimental context.

Discussion and Insights

The study highlights several critical pitfalls in machine learning benchmarking. First, the "ANOVA Trap" is real: real-world datasets often exhibit heteroscedasticity (varying noise levels), which violates ANOVA's core assumptions and necessitates non-parametric alternatives. Second, average rank is a more reliable indicator of general performance than raw Mean Squared Error (MSE), which can be skewed by the scale of individual datasets.

Finally, the authors emphasize that a model being "best" in rank does not automatically make it "significantly better" than its closest competitors. Rigorous post-hoc testing is essential to avoid overstating the advantages of new methods.

Conclusion and Future Work

This research underscores the importance of statistical rigor in online learning. While OLR-WA showed promising results, there remains significant room for improvement in handling long-term stability under extreme concept drift. Future research could extend this framework to non-linear models, such as online deep learning, and explore automated hyperparameter tuning to enhance model adaptability across diverse data streams.

Summary updated

BibTeX
@inproceedings{abushaira2024unveiling,
  author = {Mohammad Abu-Shaira and Weishi Shi},
  title = {{Unveiling Statistical Significance of Online Regression over Multiple Datasets}},
  booktitle = {MIPR},
  year = {2024},
  doi = {10.1109/mipr62202.2024.00051},
}
Figure