TL;DR
This paper introduces CRATE, a novel "white-box" language model architecture that integrates sparse coding directly into its design, unlike traditional "black-box" models. CRATE significantly improves neuron-level interpretability, making it easier to understand what individual neurons are learning, by up to 103% compared to GPT-2.
What Problem Does This Paper Address?
Large language models (LLMs) like GPT-2 have achieved remarkable success, but they are often treated as "black boxes." This lack of interpretability makes it difficult to understand why they make certain predictions, leading to challenges in areas like debugging, ensuring fairness, and preventing undesirable behaviors like hallucination. While post-hoc methods exist to analyze neuron activations, they can be imperfect and don't fundamentally change the model's internal workings. The authors aim to address this by building interpretability into the model architecture itself, specifically focusing on improving neuron-level understanding.
Methods
The core innovation is the Coding RAte TransformEr (CRATE) architecture. Unlike standard transformers that use separate attention and MLP blocks, CRATE integrates a Multi-Head Subspace Self-Attention (MSSA) block and an Iterative Shrinkage-Thresholding Algorithm (ISTA) block.
- MSSA: This block is similar to self-attention but uses identical query, key, and value projection matrices within each head. It's designed to compress token representations and align them with semantic axes.
- ISTA: This block replaces the standard MLP and is designed to enforce sparsity in the representations. It's based on an unrolled optimization process that iteratively shrinks and thresholds activations, aiming to make neurons more "mono-semantic" (each neuron represents a single concept).
Crucially, CRATE is a "white-box" model because these sparse coding mechanisms are embedded directly within its layers, rather than being applied as an afterthought. The authors also adapt the architecture for next-token prediction by incorporating a causal mask and modifying the embedding and head layers.
Datasets
- Training: The Pile dataset (uncopyrighted) was used for pre-training.
- Evaluation:
- OpenWebText
- LAMBADA
- WikiText
- Pile dataset (for interpretability evaluation)
Key Insights and Ideas
- Interpretability by Design: The paper strongly advocates for building interpretability into the model architecture from the ground up, rather than relying solely on post-hoc analysis. This "white-box" approach leads to more fundamental improvements.
- Sparsity for Clarity: By explicitly enforcing sparsity through the ISTA block, CRATE encourages neurons to become more specialized, leading to clearer and more distinct semantic representations.
- Consistent Interpretability Across Layers: A significant finding is that CRATE's improved interpretability is consistent across different layers of the model, regardless of its size. This suggests a robust mechanism for understanding.
- Trade-off Between Performance and Interpretability: While CRATE excels in interpretability, its performance on next-token prediction is slightly lower than GPT-2. This highlights a potential trade-off that future work might aim to balance.
Key Results
- Interpretability Improvement: CRATE achieves significantly higher neuron-level interpretability, with improvements of up to 103% relative to GPT-2 on certain metrics (specifically, the OpenAI Random-only metric for L=6).
- Model Size Scalability: The interpretability gains are observed across various model sizes (L=1, 2, 3, 6, 12), demonstrating the scalability of the CRATE approach.
- Steady Interpretability: CRATE shows more stable interpretability scores across layers compared to GPT-2, indicating a more consistent internal representation.
- Qualitative Examples: Figure 1 and other qualitative analyses show that CRATE neurons activate more consistently and distinctly on relevant tokens, leading to more precise explanations.
Limitations and Future Work
- Performance Gap: CRATE's performance on the next-token prediction task is slightly lower than GPT-2, potentially due to the inherent sparsity introduced by the ISTA operator.
- Trade-off Optimization: Future work should focus on finding a better balance between model performance and interpretability, possibly by refining the sparsity mechanisms or exploring alternative trade-offs.
- Downstream Applications: The authors suggest further research into how these improved white-box mechanisms can be leveraged for downstream tasks like model editing and control.
- Broader Model Architectures: Investigating how these principles can be applied to even larger and more complex language models is also a promising direction.
Summary updated