Abstract
Concise and interpretable summaries for machine learning models and learners of the 'mlr3' ecosystem. The package takes inspiration from the summary function for (generalized) linear models but extends it to non-parametric machine learning models, based on generalization performance, model complexity, feature importances and effects, and fairness metrics.
Introduction
In traditional statistical modeling, the summary() function for linear models (lm) is a cornerstone of the R workflow, providing a comprehensive snapshot of coefficients, significance, and goodness-of-fit. However, as the field has shifted toward modern machine learning (ML) with "black-box" models like Random Forests or Gradient Boosting, this unified diagnostic view has been lost. Within the mlr3 ecosystem, practitioners often navigate a fragmented landscape: using one package for performance, another for feature importance (e.g., iml), and a third for fairness (e.g., mlr3fairness). mlr3summary addresses this fragmentation by providing a consolidated, interpretable summary output for modern ML learners.
The mlr3summary Framework
The package introduces S3 methods for Learner and GraphLearner objects, creating a unified reporting pipeline. By integrating several specialized toolkits, it offers a multi-faceted view of model behavior:
- Performance Metrics: Automatically selects metrics based on the task type (e.g., RMSE for regression, AUC for classification).
- Feature Importance & Effects: Leverages
imlandfastshapto calculate Permutation Feature Importance (PFI), SHAP values, and Accumulated Local Effects (ALE). - Model Complexity: Quantifies "sparsity" and "interaction strength" to help users apply Occam’s Razor to their models.
- Fairness Auditing: Integrates
mlr3fairnessto report demographic parity or equalized odds when protected attributes are present.
Implementation and Datasets
A key example used in the package documentation is the German Credit Dataset (credit). This preprocessed version of the Statlog dataset contains 522 observations across 6 variables (age, sex, saving accounts, duration, credit amount, and risk). It is particularly useful for demonstrating classification summaries and fairness auditing, specifically using the 'sex' attribute as a protected variable.
To manage the computational cost of advanced interpretability methods like SHAP or ALE, mlr3summary supports the future framework for parallelization, allowing users to scale computations across multiple cores efficiently.
Key Insights and Results
The primary contribution of mlr3summary is its "One-Stop-Shop" philosophy. By bundling disparate metrics into a single command, it encourages a more holistic evaluation of models.
- Standardized Defaults: The package defaults to robust methods, such as ALE over PDP, to better handle correlated features—a common pitfall in ML interpretation.
- Task-Awareness: The output dynamically adjusts based on the task. For instance, it provides confusion matrices for hard-label classifiers and residual analysis for probabilistic models.
- Flexibility: Through
summary_control(), users can fine-tune the output, hiding specific sections or adjusting the granularity of feature importance displays.
Limitations and Future Directions
While powerful, the package currently has some limitations. Complexity measures like interaction strength are presently restricted to binary classification and regression. Additionally, calculating SHAP values across many resampling iterations remains computationally expensive, though parallelization mitigates this. Future work may focus on expanding complexity metrics and improving the alignment of naming conventions with underlying fairness packages.
Conclusion
mlr3summary bridges the gap between the interpretability of classical statistics and the power of modern machine learning. By providing a standardized, comprehensive summary, it empowers mlr3 users to move beyond simple accuracy and conduct more rigorous, multi-dimensional model evaluations.
Summary updated