Detecting and Approximating Redundant Computational Blocks in Neural Networks

Irene Cannistraci, Emanuele Rodolà, Bastian Rieck

Abstract

Deep neural networks often learn similar internal representations, both across different models and within their own layers. While inter-network similarities have enabled techniques such as model stitching and merging, intra-network similarities present new opportunities for designing more efficient architectures. In this paper, we investigate the emergence of these internal similarities across different layers in diverse neural architectures, showing that similarity patterns emerge independently of the dataset used. We introduce a simple metric, Block Redundancy, to detect redundant blocks, providing a foundation for future architectural optimization methods. Building on this, we propose Redundant Blocks Approximation (RBA), implemented via the TOAST (Transformer Optimization using Adaptive and Simple Transformations) framework, which identifies and approximates one or more redundant computational blocks using simpler transformations. We show that the transformation $\mathcal{T}$ between two representations can be efficiently computed in closed-form, and it is enough to replace the redundant blocks from the network. RBA reduces model parameters and time complexity while maintaining good performance. We validate our method on classification tasks in the vision domain using a variety of pretrained foundational models and datasets.

Introduction

As Transformer-based "foundation models" scale in size, their deployment on edge devices is increasingly hindered by memory and power constraints. Existing compression techniques like pruning, quantization, and knowledge distillation often require expensive retraining or fine-tuning. This work addresses the challenge by exploiting intra-network redundancy: the observation that different blocks within a single deep network often perform overlapping functions or produce highly correlated outputs. By identifying these redundant segments, we can simplify the architecture without the need for traditional backpropagation.

Methodology: The TOAST Framework

The TOAST framework identifies redundant blocks within pre-trained Transformers and replaces them with lightweight, closed-form mappings. The process involves three primary steps:

  1. Similarity Analysis: Centered Kernel Alignment (CKA) is employed to quantify the similarity between latent representations of different blocks. High CKA scores between a starting block $s$ and an ending block $e$ suggest that the intermediate layers contribute minimally to the data transformation.
  2. Block Approximation: Once a redundant segment is identified (from block $s+1$ to $e$), it is replaced by a simple transformation $\mathcal{T}$.
  3. Closed-Form Calibration: The optimal $\mathcal{T}$—typically an Identity mapping or a Linear Translator—is found using a small calibration subset (e.g., 500 samples). A least-squares optimization provides a closed-form solution for the linear mapping, ensuring computational efficiency.
The TOAST framework replaces a sequence of redundant blocks with a simple transformation $\mathcal{T}$ computed in closed-form.

Experimental Setup and Insights

The framework was validated across a wide spectrum of vision benchmarks, including MNIST, CIFAR-10/100, and ImageNet-1k, using models such as ViT, DINOv2, and DeiT-S.

Model-Centric Redundancy

A key finding is that redundancy patterns are primarily a function of the model architecture and pre-training rather than the specific dataset. As shown in Figure 2, CKA similarity matrices remain remarkably consistent across diverse datasets for a given model. This suggests that architectural optimization can be performed once and generalized across tasks.

CKA similarity matrices for DINO-B and DeiT-S across various datasets, demonstrating that redundancy patterns are model-centric.

Performance and Efficiency

TOAST achieves significant efficiency gains with minimal impact on accuracy. For instance, on ViT-L, replacing blocks 17 through 23 saved 69.3 million parameters and increased throughput from 1223 to 1578 images per second, with only a 1.16% drop in ImageNet accuracy. In some instances, such as with DEiT-S, the approximation acted as a regularizer, maintaining or even slightly improving performance while reducing complexity.

Conclusion

TOAST provides a training-free path to model compression by leveraging internal representational similarities. While currently focused on vision transformers, the principles of intra-network redundancy likely extend to other modalities like LLMs. Future work will explore finer-grained redundancies within attention heads and sub-layers to further optimize foundation models for resource-constrained environments.

Summary updated

Abstract

Deep neural networks often learn similar internal representations, both across different models and within their own layers. While inter-network similarities have enabled techniques such as model stitching and merging, intra-network similarities present new opportunities for designing more efficient architectures. In this paper, we investigate the emergence of these internal similarities across different layers in diverse neural architectures, showing that similarity patterns emerge independently of the datataset used. We introduce a simple metric, Block Redundancy, to detect redundant blocks, providing a foundation for future architectural optimization methods. Building on this, we propose Redundant Blocks Approximation (RBA), a general framework that identifies and approximates one or more redundant computational blocks using simpler transformations. We show that the transformation $\mathcal{T}$ between two representations can be efficiently computed in closed-form, and it is enough to replace the redundant blocks from the network. RBA reduces model parameters and time complexity while maintaining good performance. We validate our method on classification tasks in the vision domain using a variety of pretrained foundational models and datasets.
BibTeX
@misc{cannistraci2024detecting,
  author = {Irene Cannistraci and Emanuele Rodolà and Bastian Rieck},
  title = {{Detecting and Approximating Redundant Computational Blocks in Neural Networks}},
  year = {2024},
  eprint = {2410.04941},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2410.04941},
}
Figure