Transformer Layers as Painters

Qi Sun, Marc Pickett, Aakash Kumar Nain, Llion Jones

TL;DR

Researchers have discovered that the middle layers of large language models (LLMs) act like an assembly line of painters sharing a common "vocabulary," allowing them to be reordered, skipped, or even run in parallel with surprisingly little impact on performance. While the first and last layers are specialized and critical, the uniformity of the middle layers suggests we can trade a bit of accuracy for significant gains in inference speed.

What Problem Does This Paper Address?

Despite the ubiquity of Transformer-based models like Llama and BERT, their internal mechanics remain a "black box." We know that stacking dozens of layers works, but we don't fully understand if every layer is essential or if their specific order is set in stone.

The authors investigate the impact of removing or reorganizing information throughout the layers of a pretrained, frozen transformer. Understanding this "layer-wise" behavior is crucial for two reasons: it could lead to more efficient ways to run existing models (reducing latency) and provide a blueprint for designing more robust future architectures.

Methods

The authors propose a "Painter Analogy": imagine a series of painters on an assembly line. Each receives a canvas, adds a few strokes (via residual connections), and passes it on. Because they all use the same "vocabulary" of painting, the order can sometimes be swapped, or multiple painters can work on different parts of the image simultaneously without causing a "catastrophe."

To test this, they applied several perturbations to frozen (non-fine-tuned) models:

  • Skip: Jumping over a block of middle layers.
  • Middle Repeat: Replacing a block of layers with multiple copies of a single middle layer.
  • Reverse/Random: Running middle layers in reverse or random order.
  • Parallel: Feeding the same input to multiple middle layers simultaneously and averaging their outputs.
  • Looped Parallel: Running the parallel layers multiple times, feeding the output back into the same layers.

Datasets

The study evaluated these strategies across a diverse set of benchmarks:

  • For Llama2 (Decoder-only): ARC (science questions), HellaSwag (common sense), GSM8K (math word problems), Winogrande (logic), and LAMBADA (word prediction).
  • For BERT-Large (Encoder-only): The GLUE benchmark (including tasks like CoLA, MRPC, QNLI, and SST2).
  • Generalization: Results were also validated on Mistral-7B and Pythia-6.9B.

Key Insights and Ideas

  • The Three-Zone Architecture: Transformers aren't just a uniform stack. They consist of three distinct representation spaces: the "beginning" (input processing), the "middle" (the bulk of the work), and the "ending" (output refinement). The middle zone is the largest and most uniform.
  • Uniformity, Not Redundancy: While middle layers share a representation space (they "speak the same language"), they are not doing the same thing. Replacing layers with repeats of a single layer (Middle Repeat) was catastrophic, proving that each layer performs a unique function even if they use the same "vocabulary."
  • Reasoning vs. Semantics: Layer order matters much more for "step-by-step" reasoning tasks (like math in GSM8K) than for "semantic" tasks (like common sense in HellaSwag). Reasoning is like painting a precise architectural scene where order is vital; semantics is more like a collage where the sequence is flexible.
  • Graceful Degradation: Most perturbations led to a gradual decline in performance rather than a total collapse. This suggests that LLMs are remarkably robust to structural changes in their middle layers.

Key Results

  • Parallel Potential: Running layers in parallel (the "Parallel Layer" strategy) showed graceful degradation for most tasks. For a model like Llama2-7B, running 8 layers in parallel could theoretically halve the latency of that block with manageable accuracy loss.
  • The Power of Looping: The "Looped Parallel 3X" method (running parallel layers for three iterations) was the most successful variant, significantly outperforming simple skipping or single-pass parallelization.
  • The "Middle Repeat" Failure: This was the most damaging modification. It suggests that forcing the model to stay on the same "step" of the assembly line for too long pushes the data out of the shared representation space.
  • Scaling Consistency: These patterns held true as models scaled from 7B to 70B parameters. The "beginning" and "middle" zones grow proportionally with model size, while the "ending" zone remains fixed at just one or two layers.

Limitations and Future Work

The most significant limitation is that the models were kept frozen. The authors hypothesize that "thawing" the models—performing a small amount of fine-tuning after reordering or parallelizing layers—would likely recover much of the lost performance.

Additionally, the study focuses on the what and how but leaves the theoretical why for future work. Specifically, the authors suggest that residual connections are the likely hero here, acting as the glue that allows layers to share a common representation space. Testing these variations on models without residual connections is a logical next step to confirm this theory.

Summary updated

BibTeX
@inproceedings{sun2025transformer,
  author = {Qi Sun and Marc Pickett and Aakash Kumar Nain and Llion Jones},
  title = {{Transformer Layers as Painters}},
  booktitle = {AAAI Conference on Artificial Intelligence},
  year = {2025},
  doi = {10.1609/aaai.v39i24.34708},
}
Figure