Abstract
In this position paper, we argue that a combination of visualization and verbalization techniques is beneficial for creating broad and versatile insights into the structure and decision-making processes of machine learning models. Explainability of machine learning models is emerging as an important area of research. Hence, insights into the inner workings of a trained model allow users and analysts, alike, to understand the models, develop justifications, and gain trust in the systems they inform. Explanations can be generated through different types of media, such as visualization and verbalization. Both are powerful tools that enable model interpretability. However, while their combination is arguably more powerful than each medium separately, they are currently applied and researched independently. To support our position that the combination of the two techniques is beneficial to explain machine learning models, we describe the design space of such a combination and discuss arising research questions, gaps, and opportunities.
Introduction
As machine learning (ML) models grow in complexity, their "black box" nature has become a significant bottleneck for adoption in sensitive fields like medicine and finance. Currently, research into explainability is often siloed: one group develops sophisticated visual dashboards to show trends, while another focuses on Natural Language Generation (NLG) to describe model logic.
This paper identifies a critical gap: visualization is excellent for high-level overviews but often fails to convey specific logical constraints or causal "what-if" scenarios. Conversely, pure text can quickly overwhelm users with detail. We propose that a unified framework combining these two media creates more versatile, human-centric explanations that provide deeper insights than either medium could alone.
The Design Space for Hybrid Explanations
To bridge the gap between visualization and verbalization, we introduce a design space categorized into three pillars:
- Model Aspects for Explanation: Identifying what needs explaining, ranging from training data and model structure (white-box) to final predictions and post-hoc interpretations (black-box).
- Strategies for Explanation Generation: Methods for synthesizing information, such as Double-Encoding (reinforcing memory by showing the same info in both text and viz) and Overview and Metadata (using viz for data patterns and text for technical parameters).
- Strategies for Explanation Presentation: How the user consumes the info, including Details-on-Demand (interactive tooltips) and Agent-based Dialogue (chatbot interfaces).
Illustrative Scenarios and Results
While this framework is conceptual, its utility is demonstrated through several illustrative scenarios. For example, in a classification task using a decision tree, a hybrid system allows a user to interact directly with the model's logic. Instead of just seeing a node in a tree, the user can query the system to receive a natural language explanation of the specific features driving that decision.
Qualitative evidence suggests that text is superior for "what-if" scenarios (e.g., "If the house size were 1% smaller, the classification would change") and for providing technical context like silhouette scores in plain English, which visualizations alone cannot easily convey.
Key Insights and Future Directions
Our analysis highlights that visualization is a "spatial" medium best for summaries, while verbalization is a "temporal/logical" medium best for specific rules and causal relationships. A significant challenge remains in Content Selection—deciding which of the thousands of possible patterns are "interesting" enough to be verbalized.
Future work must address the "overload problem" to prevent textual clutter and explore Semantic Enrichment, teaching models human-readable taxonomies so they can generate more meaningful labels (e.g., identifying "Cluster 5" as "Mammals"). Rigorous user studies are now required to quantify how much these hybrid approaches improve human trust and model debugging speed compared to traditional visualization-only methods.
Summary updated