Generating Explanations for Black Box Language Models

36 concepts37 papers1 proposal

Modern deep learning models achieve remarkable accuracy across natural language processing (NLP) tasks, from text classification to natural language inference. However, this performance comes at a cost: opacity. These models operate as "black boxes," mapping inputs to outputs through millions of parameters and dense latent spaces without providing human-readable reasons for their decisions.

As highlighted in the Findings of the Association for Computational Linguistics: EACL 2023, this lack of transparency makes it difficult to trust, debug, or safely deploy models in high-stakes environments. To bridge this interpretability gap, researchers are developing methods to extract, visualize, and embed human-verifiable logic directly into deep learning architectures. These approaches broadly fall into two categories: post-hoc explanations, which dissect a model's decisions after the fact, and inherently interpretable architectures, which are designed to show their work by construction.


Post-Hoc Explanations: Dissecting the Black Box

Post-hoc interpretability methods treat a trained model as an oracle and attempt to reverse-engineer its logic. The intuition here is simple: if we cannot look directly at the weights to understand a decision, we can observe how the model responds to controlled changes in its input or environment.

Causal Perturbation

One way to understand why a model made a specific prediction is to systematically alter the input text and observe the effect on the output. In A causal framework for explaining the predictions of black-box sequence-to-sequence models, Alvarez-Melis et al. formalized this via a causal framework for complex sequence-to-sequence tasks like machine translation. By perturbing words in the source sentence and measuring the causal impact on the generated translation, the framework builds a clear map of dependencies between input and output tokens.

Hierarchical Feature Interactions

While simple perturbation tells us which individual words are important, language is inherently compositional—words combine to form phrases, and phrases combine to form sentences. Highlighting isolated words often fails to capture how a model processes figurative language or negation.

To capture this compositionality, Zheng et al. developed a model-agnostic method called HEDGE in Generating Hierarchical Explanations on Text Classification via Feature Interaction Detection. Instead of scoring words independently, HEDGE detects structural interactions between adjacent phrases. It builds a hierarchical tree showing exactly how words merge step-by-step to build up or flip the model's final prediction, making the internal logic intuitive for human verification.

      [ Final Prediction: Negative ]
                /          \
      [ Not great ]      [ but acceptable ]
        /       \
     [ Not ]  [ great ]

Inherently Interpretable Architectures: Showing the Work

Instead of trying to explain a black box after it is trained, a parallel line of research focuses on building models that are transparent by design. These architectures incorporate structural constraints that force the network to produce explicit justifications alongside their predictions.

Sparse Alignments via Optimal Transport

In tasks like text matching or natural language inference, a model must determine the relationship between two pieces of text (e.g., whether a premise entails a hypothesis). Standard deep models process both texts into dense vectors and compare them globally, hiding the specific points of comparison.

To fix this, Swanson et al. introduced a self-explaining text-matching architecture in Rationalizing Text Matching: Learning Sparse Alignments via Optimal Transport. By leveraging mathematical constraints from Optimal Transport theory, the model is forced to find a sparse, mathematically rigorous alignment between specific text spans in the two inputs. The model cannot make a prediction without explicitly pointing to the exact phrases it is matching, providing an automatic, human-readable justification.

Latent Semantic Trees

For tasks like sentiment analysis, models often struggle to explain how they handle complex linguistic phenomena like double negatives or contrastive clauses. In Interpreting Sentiment Composition with Latent Semantic Tree, Jiang et al. integrated a linguistically-informed grammar directly into a neural network as a latent variable. As the network learns to predict sentiment, it simultaneously learns to construct a semantic tree. This tree explicitly maps out how the meanings of individual words are combined compositionally, achieving high accuracy while remaining structurally transparent.

Graph-Based Label Mapping

Another strategy is to explicitly map the relationship between input features and classification targets. In LiGCN: Label-interpretable Graph Convolutional Networks for Multi-label Text Classification, Li et al. designed a heterogeneous graph network where both words and classification labels exist as nodes in the same space. By dynamically learning the edges between word nodes and label nodes, the architecture provides a built-in, visible map showing exactly which input words triggered which specific classification labels.


Grounding Models in Explicit Knowledge

A major driver of model opacity is the reliance on purely abstract, high-dimensional latent spaces. A growing subset of interpretability research argues for grounding neural networks in observable linguistic features or structured external knowledge bases.

Historically, NLP systems relied on explicit, rule-based knowledge integration, which provided clear justifications but lacked flexibility, as discussed by Emirkanian et al. in Knowledge integration in a robust and efficient morpho-syntactic analyzer for French. Modern approaches seek a middle ground: pairing the representational power of deep learning with explicit structures.

For instance, in Improving Automated Prediction of English Lexical Blends Through the Use of Observable Linguistic Features, Saunders demonstrated that incorporating explicit, observable linguistic features into machine learning models yields highly effective and inherently interpretable predictions for complex linguistic phenomena, bypassing the need for entirely uninterpretable latent spaces.

Similarly, detecting figurative language like metaphors typically leaves deep models guessing based on surface-level context clues. In FrameBERT: Conceptual Metaphor Detection with Frame Embedding Learning, Li et al. forced a RoBERTa-based architecture to learn explicit semantic frames from the FrameNet database. By forcing the model to compare a word's literal frame against its contextual frame, the system provides a clear, linguistically grounded explanation for why a given phrase is classified as metaphorical.


Why It Matters: Trust and Verification

The shift toward explainable NLP is not just about academic curiosity; it is a prerequisite for safety and reliability. As deep learning models are increasingly deployed for information extraction, relation extraction, and text mining, understanding why a model extracts a specific relation is vital for error analysis and debugging (Song et al.; Xu et al.).

By developing tools that visualize internal states—such as the interactive LSTM visualization interface designed by Loginova et al.—and engineering architectures that inherently restrict uninterpretable behavior, the NLP community is steadily closing the interpretability gap, transforming deep neural networks from opaque black boxes into transparent, verifiable partners in language understanding.

Key Foundational Work

Conference on Empirical Methods in Natural Language Processing (1996) — contributed 1 concept elements to this direction.
Knowledge integration in a robust and efficient morpho-syntactic analyzer for French (1988) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_model_interpretability_black_box_models_4813292b,
  title = {model interpretability (black box models)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-interpretability-black-box-models/rdr_4813292bccde4b43b70dbd5b7c8b7def},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-21},
}

This exact version (version 1, frozen text):

@misc{lacuna_model_interpretability_black_box_models_4813292b_v1,
  title = {model interpretability (black box models)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/model-interpretability-black-box-models/rdr_4813292bccde4b43b70dbd5b7c8b7def/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-21},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake