Towards better analysis of machine learning models: A visual analytics perspective

Shixia Liu, Xiting Wang, Mengchen Liu, Jun Zhu

Introduction

Machine learning (ML) has achieved remarkable success in diverse fields, yet many models remain "black boxes." Their internal logic is often opaque, making it difficult for developers to understand why a model fails or how to improve it. This lack of transparency leads to a tedious trial-and-error process during model development. This paper argues for a shift toward interactive model analysis, leveraging visual analytics to transform ML into a transparent, human-in-the-loop process.

The Interactive Model Analysis Framework

The authors categorize interactive model analysis into three functional pillars: Understanding, Diagnosis, and Refinement. This framework places the human expert at the center of the modeling process, using visualization to bridge the gap between complex algorithms and human intuition.

The interactive model analysis framework for understanding, diagnosis, and refinement.
  1. Understanding: Techniques aimed at revealing the inner workings of a model. This includes point-based methods like t-SNE for projecting high-dimensional activations into 2D space, and network-based methods that visualize the topology and information flow of neural networks.
  2. Diagnosis: Tools designed to pinpoint why a model fails. Examples include "Confusion Wheels" for multi-class classification and "Squares" for visualizing prediction distributions.
  3. Refinement: Systems that allow users to provide feedback to the model. This might involve selecting better training samples, adjusting feature weights, or merging/splitting topics in a topic model based on domain expertise.

Key Insights: From Black Box to Transparency

A major challenge in visual analytics is the scale of modern models. Visualizing a few neurons is straightforward, but a modern Convolutional Neural Network (CNN) contains millions of connections. The authors highlight CNNVis as a solution that employs hierarchical clustering and edge bundling to make these massive structures manageable.

Another critical insight is the importance of visualizing the "why" behind errors. Instead of just reporting an accuracy metric, visual tools can show why a specific misclassification occurred. For instance, a model might confuse a "3" with a "5" due to specific visual similarities in the training data, a pattern that becomes clear when activations are projected into a visual space.

t-SNE visualization of MNIST training, showing clusters and misclassifications.

Furthermore, the paper emphasizes mixed-initiative guidance, where the system and the human work together. The system suggests potential improvements—such as which samples are most informative to label—and the human provides the final judgment based on domain knowledge.

Results and Applications

The survey highlights several successes in the field:

  • Labeling Efficiency: One study showed that a user could achieve 97.43% accuracy by labeling only 43 carefully selected samples through an interactive visualization.
  • Deep Learning Debugging: CNNVis enabled experts to identify activation patterns and debug complex networks that were previously too complex to analyze manually.
  • Semantic Alignment: Tools like UTOPIAN allow users to refine topic models in real-time, ensuring that mathematical clusters align with human semantics.

Future Directions

The authors identify several "grand challenges" for future research:

  • Explainability by Design: Developing models that are inherently transparent rather than relying solely on post-hoc explanations.
  • Online Analysis: Moving from static snapshots to real-time monitoring of the training process to catch issues like vanishing gradients as they occur.
  • Conflict Resolution: Creating frameworks to handle disagreements between human experts and machine suggestions in mixed-initiative systems.
  • Uncertainty Quantification: Developing more rigorous methods to visualize how uncertainty propagates through the entire ML pipeline, from data processing to final visualization.

Summary updated

BibTeX
@article{liu2017towards,
  author = {Shixia Liu and Xiting Wang and Mengchen Liu and Jun Zhu},
  title = {{Towards better analysis of machine learning models: A visual analytics perspective}},
  journal = {Vis. Informatics},
  year = {2017},
  doi = {10.1016/j.visinf.2017.01.006},
}
Figure