A causal framework for explaining the predictions of black-box sequence-to-sequence models

David Alvarez-Melis, Tommi S. Jaakkola

Abstract

Modern sequence-to-sequence (seq2seq) models achieve state-of-the-art performance in tasks like machine translation but remain notoriously opaque. We present SOCRAT (Structured-Output Causal Rationalizer), a model-agnostic framework designed to explain individual predictions by identifying causal relationships between input and output tokens. By leveraging a Variational Autoencoder (VAE) for input perturbation and Bayesian causal inference, SOCRAT partitions complex dependencies into interpretable "cognitive chunks." Our evaluation across multiple datasets demonstrates SOCRAT's ability to recover alignments, detect spurious correlations, and audit commercial models for social bias.

Introduction

Deep learning has revolutionized natural language processing (NLP), yet the "black-box" nature of these models poses significant challenges for interpretability. While attention mechanisms offer some insight, they are often insufficient as complete explanations and require internal model access. This challenge is amplified in structured prediction tasks, such as seq2seq modeling, where both inputs and outputs are variable-length sequences. Existing tools like LIME, designed for simple classification, fail to capture the nuances of these complex mappings. SOCRAT addresses this gap by providing local, post-hoc explanations for any seq2seq model without requiring access to its internal weights.

Methods

The SOCRAT framework operates in three distinct stages to decompose black-box predictions into human-readable explanations.

The SOCRAT framework pipeline: Perturbation, Causal Inference, and Explanation Selection.

1. Perturbation via VAE

To probe the model's behavior, SOCRAT generates a set of perturbed versions of the original input sentence. A Variational Autoencoder (VAE) is employed to ensure these perturbations remain "on-manifold"—semantically similar to the original input and linguistically plausible.

2. Causal Inference

The perturbed inputs are processed by the black-box model. SOCRAT then uses Bayesian logistic regression to estimate the causal influence of each input token on every output token. This step produces a bipartite graph where edges represent the strength and uncertainty of these causal dependencies.

3. Explanation Selection

A raw dependency graph is often too dense for human interpretation. SOCRAT applies a robust graph partitioning algorithm (a variation of the min-cut problem) to group tokens into "cognitive chunks." These chunks represent small, self-contained sets of input and output words that explain specific components of the prediction.

Transformation from raw causal dependencies to a partitioned explanation graph.

Datasets and Evaluation

The framework was evaluated on several diverse tasks:

  • Grapheme-to-Phoneme: Using the CMU Pronouncing Dictionary (Arpabet) as a sanity check to recover known mappings.
  • Machine Translation: Utilizing WMT14 (English-German/French) to evaluate explanations for high-quality translation models.
  • Dialogue Systems: Analyzing a chatbot trained on the OpenSubtitle Corpus to identify failure modes.

Key Results

SOCRAT's performance was benchmarked against several criteria:

  • Alignment Recovery: On the Arpabet task, SOCRAT's uncertainty-aware clustering achieved performance comparable to state-of-the-art unsupervised word alignment methods.
  • Coherence Analysis: Comparisons between human and machine translations revealed that human-generated outputs exhibit higher "coherence," characterized by stronger and more consistent causal dependencies.
  • Bias and Spurious Correlation Detection: The framework successfully identified instances where models relied on biased training data. For example, it revealed how specific gender-neutral English occupations drove gendered choices in French translations. In dialogue systems, it exposed "lazy" model behavior, where generic responses were triggered by a few keywords while ignoring the broader context.
  • Commercial Auditing: SOCRAT was used to audit commercial APIs, such as Azure’s MT service, uncovering unexplained dependencies between neutral terms and gendered articles, highlighting its utility for social bias detection.

Discussion and Limitations

While SOCRAT provides a powerful tool for model-agnostic interpretability, it faces certain limitations. The requirement for multiple queries (often >100) per explanation introduces significant computational overhead. Additionally, the framework's reliance on a well-trained VAE necessitates substantial data and careful tuning. Future research may extend this causal approach beyond text to multi-modal tasks, such as image captioning, to map causal links between visual regions and generated text.

Summary updated

BibTeX
@inproceedings{alvarezmelis2017causal,
  author = {David Alvarez-Melis and Tommi S. Jaakkola},
  title = {{A causal framework for explaining the predictions of black-box sequence-to-sequence models}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2017},
  url = {https://openreview.net/forum?id=H1ZzabG_-r},
}
Figure