Autorank: A Python package for automated ranking of classifiers

Steffen HerboldJ. Open Source Softw 2020 · DOI · DBLP

Introduction

In machine learning research, comparing the performance of different models is a fundamental task. However, determining whether one model is statistically superior to another requires a rigorous statistical pipeline that is often complex and prone to error. autorank is an open-source Python package that automates this process, ensuring that researchers adhere to established best practices, such as the Demšar guidelines, with minimal effort.

The Problem: Navigating Statistical Complexity

A common pitfall in research is relying solely on raw performance metrics like accuracy. To establish meaningful results, performance must be compared across multiple datasets using paired samples. Choosing the correct test involves navigating a series of decisions: checking for normality, assessing variance homogeneity, and selecting between parametric (e.g., paired t-test) and non-parametric (e.g., Wilcoxon) methods. When multiple models are involved, the complexity increases with the need for post-hoc analysis and family-wise error correction. For many researchers, this complexity leads to "p-hacking" or the use of inappropriate tests. autorank codifies this entire decision-making process into a simplified, automated tool.

Methodology: The Automated Decision Pipeline

The core functionality of autorank is built around a standardized decision flow that determines the most appropriate statistical test based on the characteristics of the input data.

The decision flow implemented in autorank for selecting statistical tests.

When a user provides a DataFrame of results to the autorank function, the package automatically executes the following logic:

  1. Check Populations: Determines if the comparison involves two models or multiple models.
  2. Normality & Variance Testing: Runs the Shapiro-Wilk test for normality and Levene’s or Bartlett’s tests for homogeneity of variance.
  3. Test Selection: Selects the optimal test, such as the Friedman test for multiple non-normal populations or Repeated Measures ANOVA for normal populations with equal variance.
  4. Effect Size & Confidence: Calculates effect sizes (like Cohen’s $d$ or Cliff’s delta) and confidence intervals, providing more context than a simple p-value.

Demonstration and Results

The package reduces complex statistical analysis into just three lines of Python code, producing both textual reports and high-quality visualizations:

results = autorank(data)
create_report(results)
plot_stats(data)

A key output for multi-model comparison is the Critical Difference (CD) diagram. This visualization helps researchers identify which groups of classifiers are statistically indistinguishable.

A Critical Difference (CD) diagram generated by autorank, showing groups of classifiers that are not statistically distinguishable.

In the example shown above, autorank identifies that while there are significant differences overall, certain groups of classifiers (like Naive Bayes, Random Forest, and RBF SVM) perform similarly enough that they are connected by a horizontal bar, indicating they cannot be statistically distinguished from one another.

Conclusion

autorank bridges the gap between complex statistical theory and practical machine learning research. By automating test selection, reporting effect sizes, and generating standardized visualizations, it promotes more rigorous and reproducible science in the AI community. While currently focused on paired samples, its open-source nature allows for future expansion into more diverse experimental designs.

Summary updated

BibTeX
@article{herbold2020autorank,
  author = {Steffen Herbold},
  title = {{Autorank: A Python package for automated ranking of classifiers}},
  journal = {J. Open Source Softw},
  year = {2020},
  doi = {10.21105/joss.02173},
}
Figure