Ecco: An Open Source Library for the Explainability of Transformer Language Models

J AlammarACL 2021

TL;DR

Transformer language models are incredibly powerful but notoriously opaque. This paper introduces Ecco, an open-source Python library that provides a suite of tools to demystify these models by allowing researchers to interactively explore their inner workings, from input saliency to hidden states and neuron activations. Ecco aims to foster intuition and enable deeper understanding of how these complex models learn and process language.

What Problem Does This Paper Address?

Despite the remarkable success of Transformer-based language models (like BERT and GPT) in various NLP tasks, our understanding of why they work so well remains limited. This "explainability gap" hinders our ability to debug them, identify biases, and build more reliable and trustworthy AI systems. While individual techniques for probing these models exist, there's a need for a unified, user-friendly library that integrates various explainability methods and makes them accessible within a common research workflow.

Methods

Ecco offers a comprehensive toolkit for Transformer explainability, integrating several key methods:

  • Input Saliency: Ecco visualizes which input tokens are most important for generating a specific output token. This is achieved using methods like Gradient X Inputs, which calculates feature importance by backpropagating gradients of the output score with respect to the input embeddings.
  • Hidden State Analysis: The library allows for the examination of hidden states across different layers of the Transformer. It supports various Canonical Correlation Analysis (CCA) techniques (SVCCA, PWCCA, CKA) to measure the similarity between representations at different layers or between hidden states and neuron activations.
  • Neuron Activation Analysis: Ecco provides tools to capture and analyze the activations of individual neurons, particularly within the Feed-Forward Neural Network (FFNN) sublayer. This includes:
  • Probing Classifiers: Linear probes are used to predict linguistic properties (like Part-of-Speech) from FFNN representations.
  • Non-negative Matrix Factorization (NMF): NMF is employed to decompose neuron activation patterns into a smaller number of interpretable factors, revealing underlying patterns of neuron firing related to linguistic features.
  • Interactive Visualizations: A core aspect of Ecco is its rich, interactive interfaces built directly within Jupyter notebooks, leveraging D3.js for dynamic visualizations.

Datasets

The paper does not explicitly list specific datasets used for training the models themselves, as it focuses on providing tools to analyze pre-trained models. However, for the case study on probing FFNN representations for Part-of-Speech (PoS) information, the Universal Dependencies version 2 part-of-speech dataset in English was used.

Key Insights and Ideas

  • Syntactic Information in FFNNs: The paper demonstrates that FFNN representations can encode syntactic information (like singular/plural agreement) at levels comparable to hidden states, challenging the notion that FFNNs are solely for semantic processing.
  • Hidden States Build Syntactic Information: Curiously, the model appears to build up syntactic information in its hidden states even when intermediate FFNNs show diminished levels of this information, suggesting a layered and potentially complex information processing pipeline.
  • Unified Explainability Library: The core idea is that a single, integrated library like Ecco can significantly lower the barrier to entry for Transformer explainability research, making complex analysis tools more accessible.
  • Interactive Exploration is Key: The emphasis on interactive visualizations is crucial for building intuition, allowing researchers to dynamically explore model behavior rather than relying on static outputs.

Key Results

  • Successful PoS Encoding: The case study showed that FFNN neuron activations encode PoS information with accuracy comparable to hidden states. Selectivity scores indicated that this information was genuinely extracted rather than memorized.
  • Layer-Specific Information Processing: The results in Table 1 highlight interesting divergences in accuracy and selectivity scores between FFNN activations and hidden states across different layers, suggesting that layers might specialize in processing different aspects of linguistic information. For instance, while FFNN accuracy might drop in certain layers, hidden state accuracy continues to increase.
  • Ecco's Functionality: The paper showcases Ecco's ability to generate visualizations for input saliency (Figure 3 & 4), hidden state evolution (Figure 7), and token rankings (Figure 8), demonstrating its practical utility.

Limitations and Future Work

  • Input Saliency Support: Currently, Ecco's input saliency feature is primarily supported for GPT2-based models, with broader support for BERT and RoBERTa available for neuron activation collection and dimensionality reduction.
  • Community Contributions: The authors explicitly welcome community contributions to expand support for more models and explainability methods.
  • Building Testable Hypotheses: While Ecco excels at building intuition through visualization, the authors echo sentiments that this should serve as a starting point for formulating testable and falsifiable hypotheses about model interpretability.
  • Divergence in PoS Encoding: The observed divergence in PoS information encoding between FFNNs and hidden states across layers presents an avenue for future work to investigate how different layers specialize and contribute to the model's overall understanding.

Summary updated

BibTeX
@inproceedings{alammar2021ecco,
  author = {J Alammar},
  title = {{Ecco: An Open Source Library for the Explainability of Transformer Language Models}},
  booktitle = {Annual Meeting of the Association for Computational Linguistics},
  year = {2021},
}
Figure