Representation Similarity Reveals Implicit Layer Grouping in Neural Networks

Tian Gao, Amit Dhurandhar, Karthikeyan Natesan Ramamurthy, Dennis WeiMechInterp @ NeurIPS 2025 · OpenReview

TL;DR

This paper introduces a novel method using Gromov-Wasserstein (GW) distance to analyze representation similarity between neural network layers. This approach reveals implicit "layer groups" that correspond to functional abstractions, suggesting that neural network computations can exhibit abrupt shifts rather than smooth transitions. This has implications for understanding, compressing, and fine-tuning complex models.

What Problem Does This Paper Address?

As neural networks, especially large language models (LLMs), become increasingly complex and opaque, understanding their inner workings is crucial for building trustworthy and explainable AI. A common strategy is to analyze how representations evolve across network layers. However, directly comparing these intermediate representations is challenging due to varying distributions and dimensionalities. This paper tackles the problem of identifying meaningful structural boundaries and groupings within neural networks by analyzing the similarity of layer representations, aiming for a more holistic understanding of their internal computations.

Methods

The core contribution of this paper is the proposal and application of the Gromov-Wasserstein (GW) distance as a measure of representation similarity between neural network layers. Unlike standard metrics (e.g., Euclidean, cosine), GW distance is designed to compare distributions with potentially different supports and dimensions, making it suitable for intermediate representations that can vary significantly. It achieves this by finding an optimal matching between samples from two layers that preserves pairwise distances, effectively capturing structural similarities even when distributions or dimensions differ. The authors also explore other similarity measures as baselines for comparison.

Datasets

The paper validates its approach on a variety of tasks and datasets:

  • Algebraic Tasks: Synthetic "Modular Sum" tasks (fmod and fmod3) were used to test the method's ability to identify known functional partitions.
  • Natural Language Processing (NLP) Tasks:

    • Yelp reviews dataset (sentiment analysis)
    • Stanford Sentiment Treebank-v2 (SST2) dataset (sentiment analysis) These datasets were used with fine-tuned BERT models.
  • Computer Vision Tasks:

    • CIFAR-10 dataset (image classification) This dataset was used with ResNet9 models.

Key Insights and Ideas

  • Abrupt Functional Shifts: The most striking insight is that neural network computations might not be a smooth, continuous process. Instead, GW distance reveals distinct "layer groups" that suggest abrupt shifts in function or abstraction.
  • Implicit Layer Structure: The method successfully identifies meaningful groupings of layers that correspond to functional abstractions within the network, offering a way to decompose complex models into more interpretable sub-units.
  • GW Distance Superiority: GW distance consistently outperforms other similarity measures (like Euclidean, cosine, RSA, CKA) in revealing these layer structures, especially when dealing with varying distributions and dimensions.
  • Fine-tuning Focuses Later Layers: For NLP tasks, fine-tuning primarily leads to significant changes in later layers, suggesting these layers are crucial for task-specific adaptation.
  • Model Compression Potential: The identified layer groupings can inform model compression strategies, potentially allowing for the removal of redundant layers without significant performance loss.

Key Results

  • Consistent Layer Grouping: Across algebraic, NLP, and vision tasks, the GW distance consistently revealed distinct layer groups that align with functional abstractions. For instance, in modular sum tasks, it identified layers corresponding to intermediate computations.
  • NLP Task Adaptation: In fine-tuning BERT models on Yelp and SST2, GW distance highlighted that later blocks (e.g., blocks 9-12) are where most of the function/task fitting occurs.
  • Model Compression Validation: Freezing earlier layers and fine-tuning only later layers (identified by GW distance) resulted in minimal accuracy drops, demonstrating the practical utility of the approach for model compression. For example, fine-tuning only layers 8-12 on BERT achieved performance close to the full model.
  • ResNet Structure: For ResNet9 on CIFAR-10, GW distance showed clear divisions into 4 groups, while other methods provided less clear patterns.

Limitations and Future Work

  • Computational Cost: GW distance is computationally more expensive than simpler metrics, although approximate solvers are used.
  • Invariance Limitations: While GW is invariant to many transformations, there might be cases where certain invariances (like permutation invariance) could mask meaningful structural differences.
  • Access to Representations: The approach requires access to intermediate layer representations, which might not be available for all "black-box" models.
  • Future Work: The authors suggest exploring other models and applications, investigating general trends, and conducting theoretical studies on the properties of GW distances in the context of neural network interpretability. They also highlight the potential for applying this method to understand privacy and fairness issues.

Summary updated

Abstract

Providing human-understandable insights into the inner workings of neural networks is an important step toward achieving more explainable and trustworthy AI. Analyzing representations across neural layers has become a widely used approach for this purpose in various applications. In this work, we take a step toward a more holistic understanding of neural layers by investigating the existence of distinct layer groupings within them. Specifically, we explore using representation similarity within neural networks to identify clusters of similar layers, revealing potential layer groupings. We achieve this by proposing, for the first time to our knowledge, the use of Gromov-Wasserstein distance, which overcomes challenges posed by varying distributions and dimensionalities across intermediate representations--issues that complicate direct layer-to-layer comparisons. On algebraic, language, and vision tasks, we observe the emergence of layer groups that correspond to functional abstractions within networks. These results reveal implicit layer structure pattern, and suggest that the network computations may exhibit abrupt shifts rather than smooth transitions. Through downstream applications of model compression and fine-tuning, we validate our measure and further show the proposed approach offers meaningful insights into the internal behavior of neural networks.
BibTeX
@inproceedings{gao2025representation,
  author = {Tian Gao and Amit Dhurandhar and Karthikeyan Natesan Ramamurthy and Dennis Wei},
  title = {{Representation Similarity Reveals Implicit Layer Grouping in Neural Networks}},
  booktitle = {MechInterp @ NeurIPS},
  year = {2025},
  url = {https://openreview.net/forum?id=Ha5UIYfjTO},
}
Figure