Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning

Xinyuan Song, Keyu Wang, Pengxiang Li, Lu Yin, Shiwei Liu

Abstract

Recent studies suggest that the deeper layers of Large Language Models (LLMs) contribute little to representation learning and can often be removed without significant performance loss. However, such claims are typically drawn from narrow evaluations and may overlook important aspects of model behavior. In this work, we present a systematic study of depth utilization across diverse dimensions, including evaluation protocols, task categories, and model architectures. Our analysis confirms that very deep layers are generally less effective than earlier ones, but their contributions vary substantially with the evaluation setting. Under likelihood-based metrics without generation, pruning most layers preserves performance, with only the initial few being critical. By contrast, generation-based evaluation uncovers indispensable roles for middle and deeper layers in enabling reasoning and maintaining long-range coherence. We further find that knowledge and retrieval are concentrated in shallow components, whereas reasoning accuracy relies heavily on deeper layers -- yet can be reshaped through distillation. These results highlight that depth usage in LLMs is highly heterogeneous and context-dependent, underscoring the need for task-, metric-, and model-aware perspectives in both interpreting and compressing large models.

Introduction

As Large Language Models (LLMs) scale to hundreds of layers, a phenomenon termed the "Curse of Depth" has emerged. Observations of rank collapse or variance explosion in deeper layers have led to the popular practice of "layer pruning," where the final 20–30% of a model is removed with minimal reported impact on performance. This paper challenges the notion of deep-layer redundancy, arguing that such conclusions are often artifacts of narrow evaluation protocols. By mapping layer contributions across specific cognitive tasks—including memory retrieval, factual knowledge, and multi-step reasoning—we demonstrate that depth utilization is highly context-dependent.

Methodology

To investigate functional stratification across depth, we conducted a systematic ablation study using LLaMA-3.1, Qwen3, and LLaMA-1 model families. Our approach included:

  • Layer and Head Pruning: Measuring performance degradation following the removal of individual layers or specific attention heads.
  • Metric Comparison: Contrasting likelihood-based metrics (scoring fixed options) with generation-based metrics (evaluating free-form model output).
  • Delta Model Replacement: A novel technique using output projections from distilled model "deltas" to observe how training objectives reshape layer importance.

Evaluations were performed across a diverse benchmark suite, including MMLU (general knowledge), HellaSwag (commonsense), GSM8K (mathematical reasoning), and specialized KV Retrieval tasks.

Results and Discussion

The Impact of Evaluation Metrics

A primary finding is that the perceived redundancy of deep layers is heavily influenced by the choice of metric. Likelihood-based evaluations, such as multiple-choice scoring, primarily emphasize shallow representations. In these settings, deep layers appear largely expendable. However, generation-based evaluations reveal that middle and deep layers are critical for maintaining the long-range coherence and logical consistency required for complex tasks.

Comparison of MMLU performance and pruning sensitivity across different evaluation protocols for Qwen3-8B.

Functional Stratification of Layers

Our analysis reveals a clear division of labor based on model depth:

  • Shallow Layers: Act as "librarians," handling retrieval and basic contextual plausibility. In KV Retrieval tasks, removing the first two layers caused accuracy to plummet by up to 80%, whereas pruning deep layers had negligible impact.
  • Middle and Deep Layers: Function as "engineers," essential for symbolic manipulation and semantic integration. On the GSM8K math benchmark, sharp performance drops occurred when these layers were pruned, particularly during Chain-of-Thought reasoning.

The Sparsity of Reasoning

Reasoning capabilities are not uniformly distributed but are often concentrated in a small subset of specialized attention heads. For instance, in Qwen3-8B, pruning a few specific heads in Layer 35 resulted in a 60% drop in reasoning accuracy. Furthermore, we found that distillation can redistribute this reasoning capacity, making the model more resilient to pruning by spreading critical functions across more layers.

Conclusion

The "Curse of Depth" is not a universal property of LLMs but a reflection of how we measure their performance. While shallow layers are the primary drivers of retrieval and basic knowledge, deep layers remain indispensable for the reasoning and coherence that define advanced model behavior. Future work in model compression and architectural design must move beyond uniform pruning toward task-aware strategies that preserve these specialized functional "bottlenecks."

Summary updated

Abstract

Recent studies suggest that the deeper layers of Large Language Models (LLMs) contribute little to representation learning and can often be removed without significant performance loss. However, such claims are typically drawn from narrow evaluations and may overlook important aspects of model behavior. In this work, we present a systematic study of depth utilization across diverse dimensions, including evaluation protocols, task categories, and model architectures. Our analysis confirms that very deep layers are generally less effective than earlier ones, but their contributions vary substantially with the evaluation setting. Under likelihood-based metrics without generation, pruning most layers preserves performance, with only the initial few being critical. By contrast, generation-based evaluation uncovers indispensable roles for middle and deeper layers in enabling reasoning and maintaining long-range coherence. We further find that knowledge and retrieval are concentrated in shallow components, whereas reasoning accuracy relies heavily on deeper layers -- yet can be reshaped through distillation. These results highlight that depth usage in LLMs is highly heterogeneous and context-dependent, underscoring the need for task-, metric-, and model-aware perspectives in both interpreting and compressing large models.
BibTeX
@misc{song2025demystifying,
  author = {Xinyuan Song and Keyu Wang and Pengxiang Li and Lu Yin and Shiwei Liu},
  title = {{Demystifying the Roles of LLM Layers in Retrieval, Knowledge, and Reasoning}},
  year = {2025},
  eprint = {2510.02091},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2510.02091},
}
Figure