Abstract
We interpret the predictions of any black-box structured input-structured output model around a specific input-output pair. Our method returns an "explanation" consisting of groups of input-output tokens that are causally related. These dependencies are inferred by querying the black-box model with perturbed inputs, generating a graph over tokens from the responses, and solving a partitioning problem to select the most relevant components. We focus the general approach on sequence-to-sequence problems, adopting a variational autoencoder to yield meaningful input perturbations. We test our method across several NLP sequence generation tasks.
Introduction
Deep learning has fundamentally transformed Natural Language Processing (NLP), enabling state-of-the-art performance in complex tasks like machine translation and dialogue generation. However, these models—often based on recurrent neural networks (RNNs) or Transformers—frequently operate as "black boxes," offering little insight into the logic behind specific predictions. While attention mechanisms are often used as a proxy for interpretability, they primarily highlight correlations rather than causal relationships and are not universally applicable to all architectures.
The challenge of interpretability is particularly acute in structured prediction tasks, such as sequence-to-sequence (seq2seq) modeling. Unlike simple classification, where an explanation might target a single label, seq2seq models involve variable-length inputs and outputs with complex, many-to-many dependencies. Explaining why a model generated a specific word in a long sentence requires a granular understanding of which input components drove that decision. This paper introduces a model-agnostic framework designed to provide "prediction interpretability" by identifying causal links between input and output tokens.
Methodology: The SOCRAT Framework
The authors propose the Structured-Output Causal Rationalizer (SOCRAT), a framework that explains individual predictions by treating the underlying model as an oracle. SOCRAT operates through a three-stage pipeline:
- Perturbation Model: To establish causality, the system must observe how changes in the input affect the output. However, discrete text perturbations (like word deletion) often result in ungrammatical sequences that fall outside the model's training distribution. To solve this, SOCRAT employs a Variational Autoencoder (VAE). The input sentence is mapped to a continuous latent space, and points in the local neighborhood are sampled to generate semantically related sentence variations.
- Causal Inference: These perturbed inputs are fed into the black-box model. SOCRAT then uses Bayesian logistic regression to estimate the causal dependency between specific input tokens and output tokens. This process generates a weighted bipartite graph where edges represent the strength of the causal influence.
- Explanation Selection: A raw graph of dependencies can be overly dense and difficult for humans to interpret. SOCRAT applies a robust min-cut partitioning algorithm to group these dependencies into "cognitive chunks." The final output is an "explanation graph" consisting of sets of input and output tokens that are causally linked, such as a source-language noun phrase and its corresponding target-language translation.
Experimental Evaluation
The framework was evaluated across several diverse NLP tasks to test its versatility:
- Grapheme-to-Phoneme (Arpabet): Using the CMU Dictionary to verify if SOCRAT could recover known character-to-sound mappings.
- Machine Translation (WMT14): Evaluating explanations for English-German and English-French translation models.
- Dialogue Systems (OpenSubtitle): Diagnosing flaws in a "mediocre" model trained on subtitle data.
- Commercial APIs (Azure MT): Testing the framework on real-world, proprietary black boxes to detect systemic biases.
Key Results and Insights
SOCRAT demonstrated a strong ability to recover meaningful alignments. In the Arpabet task, the system recovered character-to-phoneme mappings with an alignment error rate (AER) only slightly higher than specialized, state-of-the-art unsupervised aligners, despite SOCRAT being a general-purpose tool.
Beyond simple alignment, the framework proved invaluable for model diagnostics. When applied to a dialogue system that frequently produced generic "I don't know" responses, SOCRAT revealed that the model was only causally attending to the first few words of the input, effectively ignoring the context.
Furthermore, the authors utilized SOCRAT for bias detection in commercial translation APIs. The framework "red-flagged" instances where a model's choice of gendered pronouns was causally linked to occupational terms (e.g., associating "doctor" with masculine pronouns and "nurse" with feminine ones) even when the source sentence was gender-neutral. This highlights the framework's utility in auditing black-box systems for social biases and spurious correlations, such as a model falsely associating specific discourse markers with informal registers due to training data quirks.
Conclusion
SOCRAT offers a robust, model-agnostic approach to interpretability in structured prediction. By prioritizing causality over simple correlation and providing "chunked" explanations, it aligns more closely with human reasoning. While the complexity of training VAEs for text and the computational cost of sampling remain challenges, the framework provides a powerful tool for diagnosing, auditing, and understanding the increasingly complex models that power modern NLP.
Summary updated