modelDown: automated website generator with interpretable documentation for predictive machine learning models

Kamil Romaszko, Magda Tatarynowicz, Mateusz Urbański, Przemyslaw BiecekJ. Open Source Softw 2019 · DOI · DBLP

Introduction

As machine learning models grow in complexity, they often become "black boxes" with thousands or millions of parameters. While the field of Explainable AI (XAI) has produced numerous tools to peek inside these models, using them is often a fragmented and repetitive process. Data scientists frequently find themselves running the same diagnostic scripts for every new model iteration.

modelDown is an R package that addresses this gap by automating the creation of comprehensive, interactive HTML websites for documenting and interpreting predictive machine learning models. By providing a single-command interface to generate visual explanations—ranging from feature importance to data drift—it transforms "black box" models into transparent, shareable assets for data science teams.

Methods

modelDown is built for the R programming language and leverages the existing DALEX (Descriptive mAchine Learning Explanations) ecosystem. DALEX provides a uniform "explainer" interface that allows modelDown to remain model-agnostic; it doesn't matter if the user is employing a Random Forest, a Gradient Boosted Machine, or a simple Linear Model.

The package uses a modular, template-based approach:

  • Engine: It utilizes whisker HTML templates and ggplot2 for visualizations.
  • Workflow: Users create "explainer" objects for their models and then call a single function, modelDown::modelDown().
  • Reproducibility: To ensure results can be audited later, the package uses the archivist library to save model exploration results in binary form. This allows collaborators to reload the exact state of an explanation at a later date.

Case Study: Titanic Dataset

The package's capabilities are demonstrated using the classic Titanic dataset. Several models were trained and compared, including Random Forest (via the ranger package), Generalized Linear Models (GLM), k-Nearest Neighbours (k-NN), and Support Vector Machines (SVM).

Key Insights and Results

The primary output of modelDown is a multi-tabbed static website that facilitates deep model auditing and side-by-side comparison.

Feature Importance and Response

The tool clearly visualizes feature importance across different models. In the Titanic demonstration, "gender" and "class" were identified as the dominant predictors across most high-performing models. However, the k-NN model prioritized different features, signaling potential issues in capturing underlying patterns.

Feature Importance tab in modelDown

Beyond simple importance, the tool incorporates Partial Dependency Plots (PDP) and Factor Merger Plots to explain how specific variables, such as age or fare, impact final predictions.

Model Performance and Auditing

The "Auditor" tab provides automated residual analysis. Instead of relying solely on aggregate accuracy scores, modelDown generates plots that help identify where a model consistently fails, making debugging more intuitive.

Model Performance tab showing residual distribution

Detecting Drift

Through drifter integration, the tool identifies concept or data drift. This is critical for production environments, alerting users if the live data deviates significantly from the training distribution.

Limitations and Future Work

While modelDown provides a robust framework, it is currently tied to the R language and the DALEX framework. Python users would require alternative wrappers. The package is designed to be extensible, allowing for the inclusion of new XAI tools as they emerge within the R community. As part of the broader DrWhy.AI initiative, future work involves tighter integration with automated ML (AutoML) pipelines for end-to-end model maintenance.

Summary updated

BibTeX
@article{romaszko2019modeldown,
  author = {Kamil Romaszko and Magda Tatarynowicz and Mateusz Urbański and Przemyslaw Biecek},
  title = {{modelDown: automated website generator with interpretable documentation for predictive machine learning models}},
  journal = {J. Open Source Softw.},
  year = {2019},
  doi = {10.21105/joss.01444},
}
Figure