Understanding the Inner Workings of Language Models Through Representation Dissimilarity

Davis Brown, Charles Godfrey, Nicholas Konz, Jonathan H. Tu, Henry Kvinge

TL;DR

This paper introduces representation dissimilarity measures as a powerful tool for understanding the inner workings of language models. By analyzing how internal representations differ between models, the authors uncover surprising asymmetries in activation functions, pinpoint generalization failures invisible to standard metrics, and reveal emergent structural properties in scaling language models.

What Problem Does This Paper Address?

As large language models (LLMs) become increasingly integrated into real-world applications, understanding how they work internally is crucial for trust, interpretability, and transparency. Traditional evaluation metrics often focus on performance on specific tasks, which can mask critical differences in how models learn and generalize. For instance, two models might achieve similar accuracy but learn fundamentally different features, leading to unpredictable behavior on out-of-distribution data. This paper addresses the challenge of quantifying and comparing the internal representations of LLMs to gain deeper insights into their mechanics, especially when these insights are not apparent from task performance alone.

Methods

The core of this work lies in applying and adapting representation dissimilarity measures to language models. The paper focuses on two primary techniques:

  • Model Stitching: This method, adapted from computer vision, involves taking the early layers of one model (f) and connecting them to the later layers of another model (g) via a learnable linear layer. The performance of this "stitched" model on a downstream task indicates how compatible the representations of f and g are. A low "stitching loss" suggests similar representations.
  • Centered Kernel Alignment (CKA): CKA measures the statistical similarity between the representations of two models (or two layers within a model) for a fixed set of inputs. It quantifies how similarly the structure of the learned features aligns, independent of task performance.

The authors also introduce a novel application of HSIC1 (Hilbert-Schmidt Independence Criterion) for comparing representations, particularly useful for analyzing scaling families of models.

Datasets

The paper utilizes the following datasets:

  • The Pile: A large, diverse dataset (800GB) used for training the SoLU/GeLU and Pythia models.
  • Multi-Genre Natural Language Inference (MNLI): Used for fine-tuning BERT models and evaluating generalization.
  • Heuristic Analysis for NLI Systems (HANS): An out-of-distribution dataset used to identify generalization failures in BERT models.

No new benchmarks were introduced; instead, existing datasets were used to demonstrate the efficacy of the dissimilarity measures.

Key Insights and Ideas

  • Activation Function Asymmetry: The paper reveals a surprising asymmetry between GeLU and SoLU activation functions. Stitching SoLU representations into GeLU models incurs a larger penalty than the reverse, suggesting GeLU might encode more useful information for training. This is a subtle difference not easily captured by standard performance metrics.
  • Locating Generalization Failures: Representation dissimilarity measures (both stitching and CKA) can pinpoint where in the model's layers generalization differences emerge. This is crucial because models that generalize well to out-of-distribution data can look identical to poorly generalizing models on in-distribution tasks.
  • Emergent "Block Structure" in Scaling: Similar to observations in computer vision, CKA analysis of the Pythia model family (varying in width and depth) reveals an emergent "block structure" in feature similarity across layers. This suggests that models develop distinct feature representations in early, middle, and late layers, with high similarity within these blocks and lower similarity between them.
  • Architectural Anomalies: CKA analysis of the Pythia family also highlighted an anomaly in the pythia-2.8b model, which showed significantly lower feature similarity with other models in the family, potentially due to its unusually small attention head dimension.

Key Results

  • SoLU/GeLU Asymmetry: Stitching experiments showed a consistent penalty when using SoLU representations as input to GeLU layers compared to the other way around, indicating a potential difference in representational richness.
  • Generalization Localization: Both identity stitching and CKA successfully differentiated between "generalizing" and "heuristic" BERT models on the MNLI task, showing that these differences are concentrated in the later layers of the encoder.
  • Pythia Scaling: CKA analysis of the Pythia family revealed high feature similarity between models of similar scale and depth, particularly in early and late layers, consistent with a "block structure." However, pythia-2.8b stood out with lower inter-model similarity.

Limitations and Future Work

  • Model Scale: The current experiments focus on relatively small language models. The authors acknowledge that larger models might exhibit qualitatively different features, and it's unclear if these findings will scale directly.
  • Clustering: While the paper shows that dissimilarity measures can distinguish between models (e.g., generalizing vs. heuristic BERTs), they did not attempt to use these measures for automatic clustering of models, as has been done with mode connectivity.
  • Future Work: The authors suggest exploring these techniques on larger models, applying them to a wider range of model architectures (e.g., RNNs, state-space models), and further investigating the implications of the observed "block structure" in feature similarity across layers. They also note the potential for these tools to be used for both positive (e.g., enhancing safety, robustness) and negative (e.g., extracting training data information) purposes.

Summary updated

Abstract

As language models are applied to an increasing number of real-world applications, understanding their inner workings has become an important issue in model trust, interpretability, and transparency. In this work we show that representation dissimilarity measures, which are functions that measure the extent to which two model's internal representations differ, can be a valuable tool for gaining insight into the mechanics of language models. Among our insights are: (i) an apparent asymmetry in the internal representations of model using SoLU and GeLU activation functions, (ii) evidence that dissimilarity measures can identify and locate generalization properties of models that are invisible via in-distribution test set performance, and (iii) new evaluations of how language model features vary as width and depth are increased. Our results suggest that dissimilarity measures are a promising set of tools for shedding light on the inner workings of language models.
BibTeX
@inproceedings{brown2023understanding,
  author = {Davis Brown and Charles Godfrey and Nicholas Konz and Jonathan H. Tu and Henry Kvinge},
  title = {{Understanding the Inner Workings of Language Models Through Representation Dissimilarity}},
  booktitle = {Conference on Empirical Methods in Natural Language Processing},
  year = {2023},
  doi = {10.18653/v1/2023.emnlp-main.403},
}
Figure