Traditional explainable AI (XAI) methods typically rely on feature attribution, such as saliency maps that highlight specific pixels in an image or tokens in a sentence. While these maps identify where a model is looking, they fail to explain what the model sees. This "semantic interpretability gap" creates a disconnect between low-level technical outputs and the high-level conceptual reasoning humans use to make decisions.
The Limits of Pixel-Level Attribution
Standard attribution methods often produce noisy heatmaps that are difficult to interpret. As Akula et al. argue, these low-level visual cues frequently fail to align with the high-level semantic concepts—such as "stripes" or "horns"—that humans use to categorize objects. Even when a heatmap correctly identifies a region of interest, it does not convey the semantic nature of the features within that region. For instance, Samangouei et al. note that knowing which pixels are important does not explain what specific visual transformations would be required to change a model's output.
This ambiguity places a significant "interpretation burden" on the human observer. Achtibat et al. point out that without semantic context, it is difficult to verify if a model is relying on legitimate object characteristics or irrelevant background artifacts and spurious correlations. In high-stakes fields like medicine, this gap is critical; Kumar et al. observe that clinicians cannot easily distinguish between various stages of a disease based solely on a generic attention heatmap.
Mapping Activations to Human Concepts
To bridge this gap, researchers have developed methods to translate internal model activations into human-understandable concepts. A foundational approach is Testing with Concept Activation Vectors (TCAV). The authors of TCAV propose using directional derivatives in a model's internal activation space to quantify how important a high-level concept (like "gender" or "texture") is to a prediction.
Similarly, Interpretable Basis Decomposition (IBD), proposed by Zhou et al., decomposes a network's decision into a set of human-understandable basis concepts. Instead of just showing a spatial region, IBD quantifies how much concepts like "sofa" or "table" contribute to the final classification. By moving the explanation from the input space (pixels) to a conceptual space, these methods provide a more intuitive rationale for model behavior.
Automated Concept Discovery
Manually defining concepts for every task is labor-intensive. Recent work focuses on automatically discovering these semantic units from model activations. Zhang et al. introduced Invertible Concept-based Explanations (ICE), which uses Non-negative Matrix Factorization (NMF) to identify consistent visual concepts like "bird wings" or "dog ears" without human labels.
Other frameworks seek to organize these discovered concepts into logical or structural hierarchies:
- CRAFT: Fel et al. use recursive activation factorization to decompose decisions into a hierarchy of visual concepts, identifying not just where a model looks, but the specific textures or object parts it detects.
- Visual Reasoning Explanation (VRX): Yunhao Ge proposes distilling model logic into a structural graph of concepts and their spatial relationships, allowing users to diagnose errors by seeing which object parts were missing or misinterpreted.
- Symbolic XAI: Schnake et al. move beyond heatmaps to generate human-readable logical formulas (e.g., "Feature A AND NOT Feature B") that describe the interactions between features.
Bridging to Natural Language
The ultimate goal of semantic interpretability is to provide explanations that mirror human communication. In natural language processing, Jang et al. developed KW-ATTN, which integrates external knowledge bases to allow models to focus on general concepts rather than just granular words.
In computer vision, the authors of DeViL describe a method to decode internal vision features directly into natural language descriptions. By translating activations into text, these models can explain their reasoning in a way that is immediately actionable for users. As Biran et al. suggest, framing feature contributions as "narrative roles" helps non-experts decide when to trust an AI's prediction, moving XAI from technical transparency to human-centric justification.