Visualizing Feature Effects in Machine Learning Models

32 concepts30 papers

When evaluating a complex machine learning model, global feature importance metrics tell us that a feature matters, but they fail to explain how or when it matters. To truly understand a model's internal logic, we need to trace the functional relationship between its inputs and its predictions.

Partial Dependence Analysis and Individual Conditional Expectation (ICE) curves provide a model-agnostic framework to do exactly this. By systematically perturbing features and observing the response, these techniques map out the landscape of a model's decision-making process.

Partial Dependence: The Global Average Effect

The core intuition behind a Partial Dependence Plot (PDP) is simple: if we want to isolate the effect of a single feature $X_s$ on a model's predictions, we must vary $X_s$ across its range while averaging out the effects of all other features $X_c$.

Mathematically, for a model $f(X)$, the partial dependence function is defined as:

$$\bar{f}_s(x_s) = \mathbb{E}_{X_c}[f(x_s, X_c)] \approx \frac{1}{n}\sum_{i=1}^n f(x_s, x_c^{(i)})$$

To construct this plot, we take our entire dataset and force the feature $X_s$ to take a specific value, say $x$. We then pass this modified dataset through the model and calculate the average prediction. By repeating this process for a grid of values across the range of $X_s$, we trace a curve that represents the marginal effect of that feature. As noted by Katarzyna Kobylińska et al. in their study on Lung Cancer Screening Models, this visualizes how the model's prediction changes on average as the specific feature varies.

PDPs are highly effective at revealing the structural nature of a model's learned relationships. Rather than assuming linearity, PDPs allow practitioners to see if a model has learned non-linear patterns, step functions, or plateaus. For instance:

Beyond simple visualization, this marginal analysis serves as a diagnostic tool. James Wexler et al. showed in The What-If Tool: Interactive Probing of Machine Learning Models that if a feature expected to be highly influential shows a flat line across its range, it immediately exposes training-serving skew or data pipeline failures. Similarly, Giorgio Piras et al. used partial dependence in Explaining Machine Learning DGA Detectors from DNS Traffic Data to determine whether a security model relies on simple, easily manipulated thresholds or captures robust, nuanced patterns.

Individual Conditional Expectation: Resolving the Average

While PDPs provide an excellent high-level summary, they suffer from a major vulnerability: they can lie by omission. If a feature has a strong positive effect on half of the dataset but an equally strong negative effect on the other half, the average marginal effect will be a flat line. The PDP would suggest the feature has no impact, completely obscuring the underlying dynamics.

Individual Conditional Expectation (ICE) curves solve this by disaggregating the average. Instead of plotting a single averaged line, an ICE plot draws a separate curve for each individual instance in the dataset.

As explained by Γεώργιος Παπουτσόγλου et al. in Machine learning approaches in microbiome research: challenges and best practices, ICE curves isolate the functional relationship between a feature and the prediction for individual samples. This granular view allows researchers to identify heterogeneous effects and spot distinct subgroups within the data. Ioannis Tsamardinos et al. utilized this exact mechanism in Just Add Data: automated predictive modeling for knowledge discovery and feature selection to provide instance-level insights into clinical biosignatures.

By combining PDPs and ICE curves, practitioners can perform global sensitivity analysis that respects individual variation. For example, Olivera Kotevska et al. applied this dual approach in Methodology for Interpretable Reinforcement Learning Model for HVAC Energy Control to verify that an AI control model consistently obeys thermodynamic laws across all operating scenarios, rather than just on average.

In clinical settings, this distinction is vital. Taissir Fekih Romdhane et al. demonstrated in PredictStr: A Balanced Benchmark Dataset for Improve Stroke Prediction that ICE curves expose individual risk variations that global metrics obscure, allowing for personalized risk assessment.

Mapping Interactions and Structure

When features interact strongly, single-feature PDPs and ICE curves become insufficient. To address this, the underlying framework can be extended to map multi-variable dependencies and structural interactions directly.

One approach is to scale up the partial dependence function to two dimensions, creating a surface plot. Vanya Van Belle et al. implemented this in Explaining Support Vector Machines: A Color Based Nomogram by using 2D color-based nomograms. By mapping two features simultaneously to a continuous visual field of color intensity and hue, they decomposed complex, non-linear SVM decision boundaries into interpretable pairwise contributions.

Alternatively, we can map these dependencies structurally across the entire feature space. Aleks Jakulin et al. developed a statistical framework in Testing the significance of attribute interactions that represents attributes as nodes and their irreducible dependencies as edges. This creates an interaction graph that explicitly distinguishes between synergistic and redundant feature relationships.

[Feature A] <--- Synergistic Interaction ---> [Feature B]
     \                                            /
      \---> Joint Contribution ---> [Prediction] <---/

For high-dimensional or deep learning models, Flavio Calmon et al. proposed a generalized information-theoretic approach in Generalizing Correspondence Analysis for Applications in Machine Learning. By using neural network estimators to perform correspondence analysis, they mapped complex dependencies between input spaces and classifier predictions, exposing how non-target environmental cues trigger model errors.

Critical Pitfalls: The Causal Assumption

Despite their utility, marginal effect visualizations carry a significant risk: they can be highly deceptive when features are strongly correlated.

Because PDPs and ICE curves vary a feature uniformly across a grid while keeping other variables constant, they inevitably generate synthetic data points that are physically impossible or non-existent in the true data distribution (e.g., a house with 5 bedrooms but only 200 square feet of total space).

As argued by Joshua R. Loftus et al. in Model-Agnostic Auditing: A Lost Cause?, model-agnostic auditing tools that rely purely on marginalizations can fail dramatically when they ignore underlying causal structures, selection biases, and complex feature dependencies. If a model is evaluated on impossible feature combinations, the resulting curves reflect the model's arbitrary behavior in unpopulated regions of the feature space rather than true real-world relationships.

Therefore, while partial dependence analysis is a powerful tool for diagnosing what a model has learned, it must always be interpreted with a clear understanding of the underlying data distribution and its causal constraints.

Key Foundational Work

Predicting Negative Events: Using Post-discharge Data to Detect High-Risk Patients (2016) — contributed 1 concept elements to this direction.
Testing the significance of attribute interactions (2004) — contributed 1 concept elements to this direction.
Explaining Support Vector Machines: A Color Based Nomogram (2016) — contributed 1 concept elements to this direction.
History & citationVersion 2 · 2026-10-04

Current · Permanent link

  1. Version 2 (latest) · 2026-10-04 · 30 papers
  2. Version 1 · 2026-10-04 · 29 papers
Cite this direction

Latest version (a living page):

@misc{lacuna_partial_dependence_analysis_2ff02da6,
  title = {partial dependence analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/partial-dependence-analysis/rdr_2ff02da69a9b4449ad62d14b63098821},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-19},
}

This exact version (version 2, frozen text):

@misc{lacuna_partial_dependence_analysis_2ff02da6_v2,
  title = {partial dependence analysis},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/partial-dependence-analysis/rdr_2ff02da69a9b4449ad62d14b63098821/v/2},
  note = {Version 2, frozen 2026-10-04. Accessed 2026-11-19},
}

See something wrong? Report a mistake