Abstract
Recent investigations into Large Language Models (LLMs) reveal that middle layers function as a modular assembly line. By treating layers as "painters" sharing a common representation space, researchers demonstrate that these layers can be reordered, skipped, or parallelized with minimal performance degradation. This paper explores the structural robustness of Transformers and identifies a three-zone architecture that could optimize future inference strategies.
Introduction
Despite the widespread adoption of Transformer-based models like Llama and BERT, their internal operations often remain opaque. While stacking layers is known to improve performance, the necessity and specific ordering of these layers are not fully understood. This study investigates the impact of structural perturbations on frozen pretrained models to determine if every layer is essential and if their sequence is immutable.
Methodology: The Painter Analogy
The authors conceptualize Transformer layers through a "Painter Analogy." Imagine an assembly line where each painter (layer) receives a canvas, adds specific strokes via residual connections, and passes it along. Because these painters utilize a shared "vocabulary" or representation space, the assembly line may be flexible.
To test this hypothesis, five primary perturbations were applied to the middle layers of various models:
- Skip: Bypassing a block of middle layers entirely.
- Middle Repeat: Replacing a block with multiple iterations of a single middle layer.
- Reverse/Random: Reordering layers non-sequentially.
- Parallel: Processing input through multiple layers simultaneously and averaging the output.
- Looped Parallel: Iteratively running parallel layers, feeding outputs back into the same block.
Experimental Setup
The study evaluated these strategies across diverse benchmarks and architectures:
- Decoder-only (Llama2, Mistral-7B, Pythia-6.9B): Tested on ARC, HellaSwag, GSM8K, Winogrande, and LAMBADA.
- Encoder-only (BERT-Large): Evaluated using the GLUE benchmark (CoLA, MRPC, QNLI, SST2).
Key Insights
The Three-Zone Architecture
Analysis reveals that Transformers are organized into three distinct zones:
- Beginning: Specialized for initial input processing.
- Middle: The largest zone, characterized by a uniform representation space where layers are most flexible.
- Ending: A small, fixed zone (often 1-2 layers) dedicated to final output refinement.
Uniformity vs. Redundancy
While middle layers share a "vocabulary," they are not redundant. The failure of the "Middle Repeat" strategy—where repeating a single layer caused catastrophic performance loss—proves that each layer performs a unique, necessary function within the shared space.
Reasoning vs. Semantics
The importance of layer order varies by task type. "Reasoning" tasks (e.g., math problems in GSM8K) require strict sequential processing, akin to architectural drawing. Conversely, "semantic" tasks (e.g., common sense in HellaSwag) are more resilient to reordering, resembling a collage where the exact sequence of additions is less critical.
Results and Discussion
The "Looped Parallel 3X" method emerged as the most effective perturbation, significantly outperforming simple skipping. This suggests that while layers can work in parallel, iterative refinement is beneficial. Furthermore, the "Parallel Layer" strategy demonstrated "graceful degradation," implying that for models like Llama2-7B, parallelizing 8 layers could potentially halve latency with acceptable accuracy trade-offs.
The consistent failure of "Middle Repeat" suggests that forcing a model to stay on one "step" for too long pushes data out of the established representation space. Notably, these architectural patterns remained consistent as models scaled from 7B to 70B parameters.
Conclusion
The structural robustness of LLM middle layers offers a promising path for inference optimization. While this study utilized frozen models, future work involving "thawing" and fine-tuning these perturbed architectures likely holds the key to recovering lost performance. The presence of residual connections is hypothesized to be the primary mechanism enabling this layer-wise flexibility, serving as the "glue" for the shared representation space.
Summary updated