Going beyond Visualization. Verbalization as Complementary Medium to Explain Machine Learning Models

Rita Sevastjanova, Fabian Beck, Basil Ell, Cagatay Turkay, Rafael Henkin, Miriam Butt, Daniel A. Keim, Mennatallah El-AssadyCity 2018 · OpenAlex

Abstract

In this position paper, we argue that a combination of visualization and verbalization techniques is beneficial for creating broad and versatile insights into the structure and decision-making processes of machine learning models. Explainability of machine learning models is emerging as an important area of research. Hence, insights into the inner workings of a trained model allow users and analysts, alike, to understand the models, develop justifications, and gain trust in the systems they inform. Explanations can be generated through different types of media, such as visualization and verbalization. Both are powerful tools that enable model interpretability. However, while their combination is arguably more powerful than each medium separately, they are currently applied and researched independently. To support our position that the combination of the two techniques is beneficial to explain machine learning models, we describe the design space of such a combination and discuss arising research questions, gaps, and opportunities.

Introduction

As machine learning (ML) models grow in complexity, their "black box" nature has become a significant bottleneck for adoption in sensitive fields like medicine and finance. Currently, research into explainability is often siloed: one group develops sophisticated visual dashboards to show trends, while another focuses on Natural Language Generation (NLG) to describe model logic.

This paper identifies a critical gap: visualization is excellent for high-level overviews but often fails to convey specific logical constraints or causal "what-if" scenarios. Conversely, pure text can quickly overwhelm users with detail. We propose that a unified framework combining these two media creates more versatile, human-centric explanations that provide deeper insights than either medium could alone.

The Design Space for Hybrid Explanations

To bridge the gap between visualization and verbalization, we introduce a design space categorized into three pillars:

  1. Model Aspects for Explanation: Identifying what needs explaining, ranging from training data and model structure (white-box) to final predictions and post-hoc interpretations (black-box).
  2. Strategies for Explanation Generation: Methods for synthesizing information, such as Double-Encoding (reinforcing memory by showing the same info in both text and viz) and Overview and Metadata (using viz for data patterns and text for technical parameters).
  3. Strategies for Explanation Presentation: How the user consumes the info, including Details-on-Demand (interactive tooltips) and Agent-based Dialogue (chatbot interfaces).
The proposed design space for hybrid ML explanations, mapping model aspects to generation and presentation strategies.

Illustrative Scenarios and Results

While this framework is conceptual, its utility is demonstrated through several illustrative scenarios. For example, in a classification task using a decision tree, a hybrid system allows a user to interact directly with the model's logic. Instead of just seeing a node in a tree, the user can query the system to receive a natural language explanation of the specific features driving that decision.

An illustrative example of an agent-based dialogue where a user queries a specific node in a decision tree to understand the underlying logic.

Qualitative evidence suggests that text is superior for "what-if" scenarios (e.g., "If the house size were 1% smaller, the classification would change") and for providing technical context like silhouette scores in plain English, which visualizations alone cannot easily convey.

Key Insights and Future Directions

Our analysis highlights that visualization is a "spatial" medium best for summaries, while verbalization is a "temporal/logical" medium best for specific rules and causal relationships. A significant challenge remains in Content Selection—deciding which of the thousands of possible patterns are "interesting" enough to be verbalized.

Future work must address the "overload problem" to prevent textual clutter and explore Semantic Enrichment, teaching models human-readable taxonomies so they can generate more meaningful labels (e.g., identifying "Cluster 5" as "Mammals"). Rigorous user studies are now required to quantify how much these hybrid approaches improve human trust and model debugging speed compared to traditional visualization-only methods.

Summary updated

Abstract

In this position paper, we argue that a combination of visualization and verbalization techniques is beneficial for creating broad and versatile insights into the structure and decision-making processes of machine learning models. Explainability of machine learning models is emerging as an important area of research. Hence, insights into the inner workings of a trained model allow users and analysts, alike, to understand the models, develop justifications, and gain trust in the systems they inform. Explanations can be generated through different types of media, such as visualization and verbalization. Both are powerful tools that enable model interpretability. However, while their combination is arguably more powerful than each medium separately, they are currently applied and researched independently. To support our position that the combination of the two techniques is beneficial to explain machine learning models, we describe the design space of such a combination and discuss arising research questions, gaps, and opportunities.
BibTeX
@inproceedings{sevastjanova2018going,
  author = {Rita Sevastjanova and Fabian Beck and Basil Ell and Cagatay Turkay and Rafael Henkin and Miriam Butt and Daniel A. Keim and Mennatallah El-Assady},
  title = {{Going beyond Visualization. Verbalization as Complementary Medium to Explain Machine Learning Models}},
  booktitle = {City},
  year = {2018},
}
Figure