Transformer Layers as Painters

Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones

Abstract

Recent investigations into Large Language Models (LLMs) reveal that middle layers function as a modular assembly line. By treating layers as "painters" sharing a common representation space, researchers demonstrate that these layers can be reordered, skipped, or parallelized with minimal performance degradation. This paper explores the structural robustness of Transformers and identifies a three-zone architecture that could optimize future inference strategies.

Introduction

Despite the widespread adoption of Transformer-based models like Llama and BERT, their internal operations often remain opaque. While stacking layers is known to improve performance, the necessity and specific ordering of these layers are not fully understood. This study investigates the impact of structural perturbations on frozen pretrained models to determine if every layer is essential and if their sequence is immutable.

Methodology: The Painter Analogy

The authors conceptualize Transformer layers through a "Painter Analogy." Imagine an assembly line where each painter (layer) receives a canvas, adds specific strokes via residual connections, and passes it along. Because these painters utilize a shared "vocabulary" or representation space, the assembly line may be flexible.

To test this hypothesis, five primary perturbations were applied to the middle layers of various models:

  1. Skip: Bypassing a block of middle layers entirely.
  2. Middle Repeat: Replacing a block with multiple iterations of a single middle layer.
  3. Reverse/Random: Reordering layers non-sequentially.
  4. Parallel: Processing input through multiple layers simultaneously and averaging the output.
  5. Looped Parallel: Iteratively running parallel layers, feeding outputs back into the same block.
Overview of layer perturbation strategies: (a) Skip, (b) Middle Repeat, (c) Reverse, (d) Parallel, and (e) Looped Parallel.

Experimental Setup

The study evaluated these strategies across diverse benchmarks and architectures:

  • Decoder-only (Llama2, Mistral-7B, Pythia-6.9B): Tested on ARC, HellaSwag, GSM8K, Winogrande, and LAMBADA.
  • Encoder-only (BERT-Large): Evaluated using the GLUE benchmark (CoLA, MRPC, QNLI, SST2).

Key Insights

The Three-Zone Architecture

Analysis reveals that Transformers are organized into three distinct zones:

  • Beginning: Specialized for initial input processing.
  • Middle: The largest zone, characterized by a uniform representation space where layers are most flexible.
  • Ending: A small, fixed zone (often 1-2 layers) dedicated to final output refinement.

Uniformity vs. Redundancy

While middle layers share a "vocabulary," they are not redundant. The failure of the "Middle Repeat" strategy—where repeating a single layer caused catastrophic performance loss—proves that each layer performs a unique, necessary function within the shared space.

Reasoning vs. Semantics

The importance of layer order varies by task type. "Reasoning" tasks (e.g., math problems in GSM8K) require strict sequential processing, akin to architectural drawing. Conversely, "semantic" tasks (e.g., common sense in HellaSwag) are more resilient to reordering, resembling a collage where the exact sequence of additions is less critical.

Results and Discussion

The "Looped Parallel 3X" method emerged as the most effective perturbation, significantly outperforming simple skipping. This suggests that while layers can work in parallel, iterative refinement is beneficial. Furthermore, the "Parallel Layer" strategy demonstrated "graceful degradation," implying that for models like Llama2-7B, parallelizing 8 layers could potentially halve latency with acceptable accuracy trade-offs.

Detail of the Looped Parallel strategy, which proved to be the most effective perturbation method.

The consistent failure of "Middle Repeat" suggests that forcing a model to stay on one "step" for too long pushes data out of the established representation space. Notably, these architectural patterns remained consistent as models scaled from 7B to 70B parameters.

Conclusion

The structural robustness of LLM middle layers offers a promising path for inference optimization. While this study utilized frozen models, future work involving "thawing" and fine-tuning these perturbed architectures likely holds the key to recovering lost performance. The presence of residual connections is hypothesized to be the primary mechanism enabling this layer-wise flexibility, serving as the "glue" for the shared representation space.

Summary updated

BibTeX
@inproceedings{sun2025transformer,
  author = {Qi Sun and Marc Pickett and Aakash Kumar Nain and Llion Jones},
  title = {{Transformer Layers as Painters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34708},
}
Figure