mlr3summary: Model and Learner Summaries for 'mlr3'

Susanne Dandl, Marc Becker, Bernd Bischl, Giuseppe Casalicchio, Ludwig BothmannOpenalex Unknown 2024 · DOI · OpenAlex

Abstract

Concise and interpretable summaries for machine learning models and learners of the 'mlr3' ecosystem. The package takes inspiration from the summary function for (generalized) linear models but extends it to non-parametric machine learning models, based on generalization performance, model complexity, feature importances and effects, and fairness metrics.

Introduction

In traditional statistical modeling, the summary() function for linear models (lm) is a cornerstone of the R workflow, providing a comprehensive snapshot of coefficients, significance, and goodness-of-fit. However, as the field has shifted toward modern machine learning (ML) with "black-box" models like Random Forests or Gradient Boosting, this unified diagnostic view has been lost. Within the mlr3 ecosystem, practitioners often navigate a fragmented landscape: using one package for performance, another for feature importance (e.g., iml), and a third for fairness (e.g., mlr3fairness). mlr3summary addresses this fragmentation by providing a consolidated, interpretable summary output for modern ML learners.

The mlr3summary Framework

The package introduces S3 methods for Learner and GraphLearner objects, creating a unified reporting pipeline. By integrating several specialized toolkits, it offers a multi-faceted view of model behavior:

  • Performance Metrics: Automatically selects metrics based on the task type (e.g., RMSE for regression, AUC for classification).
  • Feature Importance & Effects: Leverages iml and fastshap to calculate Permutation Feature Importance (PFI), SHAP values, and Accumulated Local Effects (ALE).
  • Model Complexity: Quantifies "sparsity" and "interaction strength" to help users apply Occam’s Razor to their models.
  • Fairness Auditing: Integrates mlr3fairness to report demographic parity or equalized odds when protected attributes are present.
The summary.Learner function accepts various arguments to customize the depth and focus of the generated report.

Implementation and Datasets

A key example used in the package documentation is the German Credit Dataset (credit). This preprocessed version of the Statlog dataset contains 522 observations across 6 variables (age, sex, saving accounts, duration, credit amount, and risk). It is particularly useful for demonstrating classification summaries and fairness auditing, specifically using the 'sex' attribute as a protected variable.

To manage the computational cost of advanced interpretability methods like SHAP or ALE, mlr3summary supports the future framework for parallelization, allowing users to scale computations across multiple cores efficiently.

Key Insights and Results

The primary contribution of mlr3summary is its "One-Stop-Shop" philosophy. By bundling disparate metrics into a single command, it encourages a more holistic evaluation of models.

  • Standardized Defaults: The package defaults to robust methods, such as ALE over PDP, to better handle correlated features—a common pitfall in ML interpretation.
  • Task-Awareness: The output dynamically adjusts based on the task. For instance, it provides confusion matrices for hard-label classifiers and residual analysis for probabilistic models.
  • Flexibility: Through summary_control(), users can fine-tune the output, hiding specific sections or adjusting the granularity of feature importance displays.

Limitations and Future Directions

While powerful, the package currently has some limitations. Complexity measures like interaction strength are presently restricted to binary classification and regression. Additionally, calculating SHAP values across many resampling iterations remains computationally expensive, though parallelization mitigates this. Future work may focus on expanding complexity metrics and improving the alignment of naming conventions with underlying fairness packages.

Conclusion

mlr3summary bridges the gap between the interpretability of classical statistics and the power of modern machine learning. By providing a standardized, comprehensive summary, it empowers mlr3 users to move beyond simple accuracy and conduct more rigorous, multi-dimensional model evaluations.

Summary updated

Abstract

Model and Learner Summaries for 'mlr3' Version 0.1.0Description Concise and interpretable summaries for machine learning models and learners of the 'mlr3' ecosystem.The package takes inspiration from the summary function for (generalized) linear models but extends it to non-parametric machine learning models, based on generalization performance, model complexity, feature importances and effects, and fairness metrics.
BibTeX
@inproceedings{dandl2024mlr,
  author = {Susanne Dandl and Marc Becker and Bernd Bischl and Giuseppe Casalicchio and Ludwig Bothmann},
  title = {{mlr3summary: Model and Learner Summaries for 'mlr3'}},
  booktitle = {Openalex Unknown},
  year = {2024},
  doi = {10.32614/cran.package.mlr3summary},
}
Figure