Abstract
In practical applications of machine learning, it is necessary to look beyond standard metrics such as test accuracy in order to validate various qualitative properties of a model. Partial dependence plots (PDPs) are a popular tool for this purpose, but they are limited by manual feature selection and axis-alignment. We propose Automated Dependence Plots (ADP), a framework that automates the search for "interesting" model behaviors by optimizing utility functions over arbitrary directions in feature or latent space. ADP helps practitioners identify non-monotonicity, bias, and out-of-sample failures that traditional accuracy metrics completely miss.
Introduction: The Limitations of Traditional Validation
In machine learning, high test accuracy is often a "leaky" metric—it indicates that a model works on average but can hide dangerous edge cases, biases, or nonsensical behaviors. To peek inside the "black box," practitioners often use Partial Dependence Plots (PDPs), which visualize how a model’s prediction changes as a single feature varies. However, traditional PDPs suffer from two major flaws:
- The "Haystack" Problem: In high-dimensional datasets, the number of potential plots to inspect is overwhelming, making manual discovery of interesting behaviors nearly impossible.
- Axis-Alignment: Standard PDPs only vary one feature at a time, missing critical interactions where a model behaves strangely only when multiple features change simultaneously.
Methodology: Automating the Search for Utility
Automated Dependence Plots (ADP) treat the selection of a visualization as an optimization problem. Instead of manual inspection, ADP defines "utility" functions to mathematically identify what makes a plot "interesting."
Utility Measures
- Model Contrast: Identifies plots where the model deviates most from a baseline, such as a linear model or a local explanation method like LIME.
- Functional Property Validation: Searches for plots that violate expected domain rules, such as monotonicity (e.g., "increasing income should not decrease a credit score") or smoothness (Lipschitz continuity).
Directional and Latent Exploration
ADP generalizes PDPs to show model responses along arbitrary directional vectors, capturing complex feature interactions. For unstructured data like images or text, ADP can optimize directions within the latent space of a generative model (e.g., a VAE) to reveal how semantic concepts, such as the rotation of a traffic sign, affect predictions.
To solve these optimization problems efficiently and without requiring model gradients, the authors introduced the Greedy Coordinate Pairs (GCP) algorithm. GCP finds sparse, human-interpretable directions that maximize the chosen utility function.
Key Results
Model Selection and Monotonicity
The paper demonstrates that models with nearly identical test accuracy can exhibit vastly different qualitative behaviors. In a loan application dataset, while Random Forest, Gradient Boosting, and Deep Neural Networks (DNN) all achieved similar accuracy, ADP revealed that the DNN exhibited wild, non-monotonic swings.
As shown in the figure above, the "Least Monotonic Utility" score quantitatively proves that the Gradient Boosting model (0.068) is more "sensible" for deployment than the DNN (2.4), despite their similar predictive performance.
Bias Detection
ADP is particularly effective for auditing models for fairness. In recidivism prediction models, ADP automatically surfaced plots showing that the effect of "Time Served" on risk scores varied significantly based on an individual's race.
This automated discovery highlights nuances that global bias metrics often overlook, providing a more granular view of how protected attributes influence model behavior.
Discussion and Limitations
While ADP provides a powerful framework for model validation, it faces certain challenges. One primary concern is the Manifold Problem: moving along arbitrary directions may result in "impossible" feature combinations (e.g., a 20-year-old with 40 years of work experience). The authors currently use Gaussian density estimates to bound plots, but more sophisticated density models may be required for complex data.
Additionally, there is an Interpretability Trade-off. While ADP can find directions involving many features, plots involving more than two or three simultaneous changes become difficult for humans to parse. Future work will involve user studies to determine how data scientists can best integrate these automated suggestions into real-world workflows.
Summary updated