TL;DR
Foundation vision models frequently compute near-duplicate representations across their depth, but simply dropping these layers causes catastrophic performance drops. This paper introduces Block Redundancy (BR) to detect these redundant blocks and Redundant Blocks Approximation (RBA) to replace entire sequences of transformer layers with a single, closed-form linear transformation. The result is a parameter- and compute-reduction technique that requires zero gradient-based fine-tuning or retraining.
What Problem Does This Paper Address?
Modern Vision Transformers (ViTs) achieve strong performance at the expense of steep parameter counts and high computational latency. Standard compression techniques—such as layer pruning, early exiting, or knowledge distillation—typically demand expensive retraining, specialized sub-networks, or iterative pruning schedules. Conversely, simply dropping ("skipping") layers in a pre-trained network without retraining degrades performance severely.
Prior work established that neural networks exhibit high internal representation similarity (layers doing roughly similar work), but few methods leverage this intra-network similarity to compress models without retraining. The authors investigate:
- Which computational blocks are actually redundant?
- Can we bypass these blocks with trivial, lightweight mathematical mappings rather than running full transformer blocks or retraining the network?
Methods
The authors propose a two-stage, training-free framework called Redundant Blocks Approximation (RBA):
-
Redundancy Identification via Block Redundancy (BR):
For any block $b$, the BR score is defined as the negative Mean Squared Error (MSE) between its output representation and the preceding block's output over a small calibration set $\mathcal{D}_{\text{sub}}$: $$\text{BR}(b) = - \frac{1}{|\mathcal{D}_{\text{sub}}|} \sum_{x \in \mathcal{D}_{\text{sub}}} \|h^{(b)}(x) - h^{(b-1)}(x)\|_2^2$$ Higher scores indicate that a block barely alters the latent space, signaling redundancy. -
Closed-Form Block Approximation:
Instead of pruning or dropping consecutive or non-consecutive blocks between layer $b_i$ and layer $b_{i+n}$, RBA replaces them with a single linear operator $T$. This transformation is solved in closed-form via ordinary least squares: $$T = \arg\min_T \|Y - T(X)\|_2^2$$ where $X$ and $Y$ are the output matrices of blocks $b_i$ and $b_{i+n}$ computed over a tiny subset (3,000 samples). This mapping is applied identically across all token positions, bypassing multi-head self-attention and MLP layers with minimal arithmetic overhead and zero backpropagation.
Datasets
The authors evaluate RBA across vision benchmarks:
- ImageNet-1k (1,000 classes)
- CIFAR-10 and CIFAR-100 (both coarse 20-class and fine 100-class splits)
- MNIST and Fashion-MNIST (F-MNIST)
Tested architectures include pre-trained ViT-S, ViT-B, DiNO-S, DiNO-B, and DEiT-S. No new benchmarks were created.
Key Insights and Ideas
- Architecture, Not Data, Dictates Redundancy Patterns: Pairwise similarity matrices across layers reveal distinct structural "fingerprints" per model family that remain virtually identical whether probed with MNIST digits or natural CIFAR images.
- Redundancy Profiles Differ by Model:
- In standard supervised ViT and self-supervised DiNO, representation change accelerates sharply in the final layers, meaning early-to-intermediate blocks are highly redundant while the final block is essential.
- In DEiT (distillation-trained), the trend reverses: middle-to-late blocks exhibit high redundancy, making the late blocks prime candidates for approximation.
- Approximation Vastly Outperforms Layer Skipping: Directly removing blocks causes severe degradation, whereas inserting an analytical linear bridge preserves feature manifold geometry (verified via PCA).
- Linear Outperforms Non-Linear MLPs: Fitting non-linear translators (e.g., multi-layer perceptrons or Res-MLP blocks with gradient descent) yielded little to no gain over the closed-form linear solution for redundant layers, despite the extra training cost.
- Cross-Dataset Generalization: Linear maps fitted on CIFAR-10 transfer seamlessly to CIFAR-100 without recalculation.
Key Results
- ImageNet-1k (ViT-S):
- Uncompressed baseline: 21.82M parameters, 73.98% accuracy.
- Bypassing 1 block ($2 \to 3$): 20.19M parameters, retaining 71.74% accuracy.
- Bypassing 3 high-BR blocks ($2 \to 5$): Reduces parameters to 16.94M and retains 60.41% accuracy. By comparison, approximating 3 low-BR blocks ($7 \to 10$) with the same parameter count plummets to 33.77%, validating the BR metric's guidance.
- RBA vs. Skipping (CIFAR-10 / ViT-S):
- Replacing blocks $1 \to 5$ via RBA achieves 84.93% accuracy (15.31M parameters), whereas naive block skipping achieves only 58.08%.
- Replacing block $2 \to 4$ achieves 93.03% with RBA vs. 71.56% with skipping.
- ViT-B Scaling (CIFAR-100 Fine):
- Approximating block $3 \to 4$ reduces parameters from 86.40M to 79.90M while accuracy remains at 82.50% (baseline: 83.41%).
Limitations and Future Work
- Domain and Architecture Scope: Experiments are restricted to vision transformers for classification; the authors have not yet demonstrated applicability to convolutional nets (ResNets), autoregressive language models, or generative/reconstruction tasks.
- Approximating Distant Blocks: When approximating large spans of non-redundant blocks (e.g., $1 \to 5$ on complex data), linear approximations struggle, and non-linear translators begin to show an advantage.
- Future Directions: Incorporating topological data analysis (TDA) metrics to better map multi-scale latent geometries and extending RBA to cross-modal architectures.
Summary updated