Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models

Jiawei Zhang, Yang Wang, Piero Molino, Lezhi Li, David S. Ebert

Abstract

Interpretation and diagnosis of machine learning models have gained renewed interest in recent years with breakthroughs in new approaches. We present Manifold, a framework that utilizes visual analysis techniques to support interpretation, debugging, and comparison of machine learning models in a more transparent and interactive manner. Conventional techniques usually focus on visualizing the internal logic of a specific model type (i.e., deep neural networks), lacking the ability to extend to a more complex scenario where different model types are integrated. To this end, Manifold is designed as a generic framework that does not rely on or access the internal logic of the model and solely observes the input (i.e., instances or features) and the output (i.e., the predicted result and probability distribution). We describe the workflow of Manifold as an iterative process consisting of three major phases that are commonly involved in the model development and diagnosis process: inspection (hypothesis), explanation (reasoning), and refinement (verification). The visual components supporting these tasks include a scatterplot-based visual summary that overviews the models' outcome and a customizable tabular view that reveals feature discrimination. We demonstrate current applications of the framework on the classification and regression tasks and discuss other potential machine learning use scenarios where Manifold can be applied.

Introduction

As machine learning (ML) models grow in complexity, they increasingly become "black boxes." While global metrics like accuracy or F1-score provide a high-level view of performance, they often fail to explain why a model fails on specific instances or how different architectures (e.g., a Random Forest vs. a Deep Neural Network) differ in their decision-making logic.

Most existing interpretation tools are model-specific, focusing on internal structures like CNN neurons or decision tree branches. However, production environments often require comparing entirely different model types. Manifold addresses this gap by providing a unified, model-agnostic workflow for inspection, explanation, and refinement.

The Manifold Framework

Manifold treats ML models as black boxes, relying solely on input features and output probabilities. The framework facilitates an iterative debugging process through two primary interactive components.

Model Comparison Overview

The comparison view utilizes a "small multiple" design consisting of a matrix of scatterplots. Each cell in the matrix compares a pair of models ($M_i$ and $M_j$). The X and Y axes represent the prediction confidence of each model, respectively. This layout naturally segments data into quadrants based on agreement and disagreement.

The Model Comparison Overview uses a matrix of scatterplots to visualize performance differences between model pairs across different classes.

In this view, users can identify "symptom sets"—specific clusters of problematic data points where models are either confidently wrong or significantly disagree. By lasso-selecting these points, users can initiate a deeper investigation into the underlying causes of these errors.

Feature Interpretation View

Once a symptom set is selected, Manifold transitions to the Feature Interpretation View. This tabular interface compares the feature distributions of the selected points against the rest of the dataset. It employs Kullback-Leibler (KL) divergence to rank features based on how much they contribute to the discrepancy between the subsets.

The Feature Attribution view ranks features by their contribution to the discrepancy between a selected data segment and the rest of the dataset.

For example, if a model consistently fails on short text inputs, this view would highlight "text length" as a high-divergence feature, providing a concrete lead for feature engineering or data augmentation.

Case Studies and Results

The authors evaluated Manifold across two distinct tasks to demonstrate its versatility:

  1. Multi-class Classification: Using the Spooky Author Identification dataset, the framework helped identify that an LSTM model was over-relying on specific frequent words (e.g., "old"), leading to misplaced confidence. Comparing it to a Naive Bayes model revealed a lack of complementarity, suggesting that these specific models might not form an effective ensemble.
  2. Regression: In the Bike Sharing Demand dataset, Manifold identified that "hour of the day" and "temperature" were high-divergence features in an "under-prediction" symptom set. By refining the feature encoders for these variables, the user successfully reduced the Root Mean Squared Logarithmic Error (RMSLE) from 0.357 to 0.341.

Discussion and Future Work

While Manifold provides a powerful model-agnostic diagnostic tool, it faces certain challenges. Domain experts noted a learning curve associated with the Cartesian-based scatterplots, where negative coordinates represent incorrect predictions. Additionally, as the number of models increases, the small-multiple matrix can become visually cluttered.

Future work will focus on automating the detection of symptom sets through clustering and addressing the potential risk of overfitting that arises when developers iteratively refine models based on granular validation-set symptoms. Despite these challenges, Manifold represents a significant step toward making complex machine learning systems more transparent and easier to debug.

Summary updated

Abstract

Interpretation and diagnosis of machine learning models have gained renewed interest in recent years with breakthroughs in new approaches. We present Manifold, a framework that utilizes visual analysis techniques to support interpretation, debugging, and comparison of machine learning models in a more transparent and interactive manner. Conventional techniques usually focus on visualizing the internal logic of a specific model type (i.e., deep neural networks), lacking the ability to extend to a more complex scenario where different model types are integrated. To this end, Manifold is designed as a generic framework that does not rely on or access the internal logic of the model and solely observes the input (i.e., instances or features) and the output (i.e., the predicted result and probability distribution). We describe the workflow of Manifold as an iterative process consisting of three major phases that are commonly involved in the model development and diagnosis process: inspection (hypothesis), explanation (reasoning), and refinement (verification). The visual components supporting these tasks include a scatterplot-based visual summary that overviews the models' outcome and a customizable tabular view that reveals feature discrimination. We demonstrate current applications of the framework on the classification and regression tasks and discuss other potential machine learning use scenarios where Manifold can be applied.
BibTeX
@article{zhang2018manifold,
  author = {Jiawei Zhang and Yang Wang and Piero Molino and Lezhi Li and David S. Ebert},
  title = {{Manifold: A Model-Agnostic Framework for Interpretation and Diagnosis of Machine Learning Models}},
  journal = {IEEE Transactions on Visualization and Computer Graphics},
  year = {2018},
  doi = {10.1109/tvcg.2018.2864499},
}
Figure