ceterisParibus: Ceteris Paribus Profiles

Przemysław BiecekOpenalex Unknown 2018 · DOI · OpenAlex

Abstract Ceteris Paribus Profiles (What-If Plots) are designed to present model responses around selected points in a feature space, such as around a single prediction for an interesting observation. These plots are model-agnostic, meaning they work for any predictive machine learning model and facilitate direct model comparisons. Ceteris Paribus Plots are intended to supplement additive attribution methods like those found in the 'breakDown' package.

TL;DR

The ceterisParibus package provides a model-agnostic framework for "What-If" analysis, allowing practitioners to peek under the hood of complex machine learning models. By generating Ceteris Paribus (all else being equal) profiles, it visualizes how a specific prediction would change if a single feature were modified, offering a powerful tool for local interpretability and model comparison.

What Problem Does This Paper Address?

As machine learning models—particularly "black boxes" like deep neural networks or ensemble methods—become more prevalent in high-stakes decision-making, the need for interpretability grows. While global explanations (like overall feature importance) tell us how a model behaves on average, they often fail to explain why the model made a specific prediction for a specific individual.

The ceterisParibus package addresses this gap by focusing on local, instance-level explanations. It answers the fundamental human question: "What would have happened if this one variable was different?" For example, if a loan application was rejected, a Ceteris Paribus plot can show exactly how much the applicant's income would need to increase to change the model's decision, holding all other factors constant.

Methods

The package implements several key techniques for local model exploration, primarily utilizing the DALEX (Descriptive mAchine Learning Explanations) ecosystem to ensure model-agnosticism. The core of the package is the ceteris_paribus() function, which defines the primary interface for generating these profiles.

Core parameters for generating Ceteris Paribus profiles.

Key methodologies include:

  • Ceteris Paribus Profiles: For a selected observation, the tool varies one feature across a range of values (a grid) while keeping all other features fixed at their original values. The model’s predictions are then plotted against this range.
  • Local Conditional Expectation (LCE): A variation of the profile that incorporates information from "neighboring" observations to provide a more context-aware local view. This is handled by the local_conditional_expectations() function.
Parameters for calculating Local Conditional Expectations (LCE).
  • Oscillations: To quantify local variable importance, the package calculates "oscillations." If a prediction changes drastically when a feature is varied, that feature has high local importance. If the profile is a flat line, the feature is irrelevant for that specific prediction.
  • 2D What-If Plots: These extend the 1D profiles to two variables, allowing researchers to visualize local interactions between features.

Datasets

As this is a technical manual for an R package, the datasets used are pedagogical examples intended to demonstrate functionality:

  • Apartments Dataset: Used to predict apartment prices based on features like surface area, floor, number of rooms, and district.
  • HR Dataset: Used to model employee turnover ("fired" vs. "not fired") based on variables like gender, age, and evaluation scores.

Key Insights and Ideas

  • Local vs. Global Importance: A variable might be globally important but have zero impact on a specific outlier. ceterisParibus makes these discrepancies visible.
  • Model Comparison via Overlay: One of the most powerful features is the ability to overlay profiles from different models for the same observation, revealing where their logic diverges in the feature space.
  • Visualizing the "Wiggle": The concept of "oscillations" provides a bridge between qualitative visualization and quantitative importance, turning visual "wiggliness" into a rankable metric.
  • S3 Extensibility: The package's S3 generic structure allows it to be extended to non-standard data sources like H2O or external databases.

Key Results

The package provides a suite of functions that produce high-quality, ggplot2-based visualizations. Key capabilities include generating individual profiles, aggregated profiles (comparing a case to the norm), and residual visualizations. Using ceteris_paribus_layer, users can build complex plots incrementally, adding rugs, points, and different model profiles to a single visualization.

Limitations and Future Work

The Ceteris Paribus approach faces theoretical challenges, particularly with correlated features. The "all else being equal" assumption can result in unrealistic data points if features are highly dependent. Additionally, calculating profiles for many observations can be computationally expensive. The author notes that this package is intended to supplement the breakDown package, suggesting that Ceteris Paribus profiles are just one piece of the broader interpretability puzzle.

Summary updated

Abstract

Ceteris Paribus Profiles (What-If Plots) are designed to present model responses around selected points in a feature space. For example around a single prediction for an interesting observation. Plots are designed to work in a model-agnostic fashion, they are working for any predictive Machine Learning model and allow for model comparisons. Ceteris Paribus Plots supplement the Break Down Plots from 'breakDown' package.
BibTeX
@inproceedings{biecek2018ceterisparibus,
  author = {Przemysław Biecek},
  title = {{ceterisParibus: Ceteris Paribus Profiles}},
  booktitle = {Openalex Unknown},
  year = {2018},
  doi = {10.32614/cran.package.ceterisparibus},
}
Figure