The Semantic Peak of Intermediate Transformer Layers

81 concepts63 papers1 proposal

We naturally expect deep neural networks to build increasingly rich, abstract representations as information flows from the input layer to the output layer. In many applications, practitioners treat the final layer of a Transformer as the definitive summary of an input's meaning.

However, a robust body of evidence reveals a striking paradox: the final layer of a deep network is frequently not the most semantically informative. Instead, general semantic utility tends to peak in the middle or penultimate layers and degrade as it approaches the terminal output. As Oscar Skean et al. demonstrate, the common practice of using a model's final layer for embeddings lacks empirical justification. Selecting an optimal intermediate layer consistently yields measurable performance gains in classification, clustering, and re-ranking tasks.

To design better embedding models and retrieval systems, we must understand the underlying mechanics of why deep representations degrade and how intermediate layers preserve a more generalized semantic essence.

Why the Last Layer Fails: Objective Over-Specialization

The primary driver behind the decay of final-layer representations is objective over-specialization. Deep transformer models are trained on specific pretext tasks—such as next-token prediction for causal Large Language Models (LLMs) or masked language modeling for bidirectional models like BERT.

As representations travel through the final few layers of the network, the model shifts its focus away from broad, global context extraction and toward the highly localized mechanics required to satisfy its training objective.

[Input Text] ---> [Early Layers: Surface Features] ---> [Middle Layers: Broad Semantics] ---> [Final Layers: Next-Token Prediction]
                                                                   ^                                         ^
                                                           (Semantic Peak)                           (Over-Specialized)

Zifeng Cheng et al. and Qing Gu et al. argue that this over-specialization introduces a task-specific bias. The final layer becomes preoccupied with vocabulary distributions and local token dynamics, which actively degrades the general utility of the embedding for downstream similarity and retrieval tasks.

This phenomenon is not unique to natural language processing; it is a fundamental characteristic of autoregressive and self-supervised architectures across modalities:

  • Vision: In pixel-level generative models like Image GPT, Alec Radford et al. found that semantic representation quality peaks in the middle layers. The final layers specialize in local pixel reconstruction, while intermediate layers retain the broader global context necessary for image classification.
  • Masked Language Models: Mengjie Zhao et al. demonstrate that the final layers of pre-trained language models specialize so heavily in masked token prediction that they become less robust for general semantic tasks than middle layers.
  • Audio: For music representation learning, the authors of Efficient Supervised Training of Audio Transformers for Music Representation Learning note that extracting representations from middle blocks yields more transferable features because they capture a cleaner balance of local and global semantic information.

The Threat of Token Homogeneity

Beyond objective specialization, deep layers suffer from a structural bottleneck known as token homogeneity. Inside a Transformer, the self-attention mechanism continuously mixes information across tokens. Over many layers, this repeated mixing can cause the representations of individual words to blend together completely, making them mathematically indistinguishable from one another.

L. Yuan et al. identify token homogeneity as the primary culprit behind why deep layers fail at tasks requiring explainability and selective rationalization. If all tokens share nearly identical representations in the final layers, it becomes impossible to isolate which specific words triggered a model's prediction. Intermediate layers provide a superior balance: they have processed enough layers to understand abstract contextual meaning, but they retain enough token heterogeneity to keep individual word semantics distinct.

Preserving Language Neutrality and Lexical Core

The tendency of final layers to over-specialize also disrupts cross-lingual transfer. In multilingual models, intermediate layers act as a language-independent space where sentences with identical meanings map closely together, regardless of the language they are written in.

As representations move into the deepest layers, they begin to re-specialize in the specific vocabulary and structural constraints of the target language's pre-training objective. Jindrich Libovický et al. observe that middle layers provide the most effective semantic representations for cross-lingual tasks because the final layers lose general semantic neutrality. To exploit this, Félix Gaschi et al. show that extracting representations from intermediate stages (such as the second-to-last layer) leverages the model's natural cross-lingual alignment before task-specific specialization degrades the transfer vector.

For fine-grained lexical tasks, Jey Han Lau demonstrates that middle layers better preserve the core lexical meaning of words across different eras and contexts, whereas final layers over-specialize on token prediction at the expense of generalized semantic essence.

Exploiting the Middle Layers

Recognizing that the "sweet spot" of a transformer lies in its intermediate states, researchers have developed several strategies to harvest these representations effectively:

Multi-Layer Pooling

Rather than guessing a single optimal layer, some approaches aggregate information across the network's depth. Yixuan Tang et al. introduced Multi-Layers Trainable Pooling, a method that uses cross-attention to dynamically gather semantic information from all internal layers. This approach outperforms standard last-layer embeddings by capturing general semantics before they are filtered out by next-token prediction objectives.

Inference-Time Steering and Prepending

Other techniques manipulate intermediate states directly during inference to bypass final-layer degradation. For instance, the Token Prepending method introduced by Qing Gu et al. takes the sentence embedding from an intermediate layer and feeds it back into the start of the next layer, forcing the model to integrate global context early on rather than relying on deep, over-specialized layers.

By shifting our focus away from the final layer and targeting the intermediate semantic peak, we can extract representations that are more robust, generalizable, and aligned with true conceptual meaning.

Key Foundational Work

Fill the GAP: Exploiting BERT for Pronoun Resolution (2019) — contributed 1 concept elements to this direction.
Cross Modal Distillation for Supervision Transfer (2016) — contributed 1 concept elements to this direction.
History & citationVersion 1 · 2026-10-04

Current · Permanent link

View frozen version 1

Cite this direction

Latest version (a living page):

@misc{lacuna_optimal_layer_selection_intermediate_lay_c3ab5c96,
  title = {optimal layer selection (intermediate layer utility)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-layer-selection-intermediate-layer-utility/rdr_c3ab5c9629114eaf9ead651030ee568f},
  note = {Latest version of a living research direction; cite a version for a fixed text. Accessed 2026-11-20},
}

This exact version (version 1, frozen text):

@misc{lacuna_optimal_layer_selection_intermediate_lay_c3ab5c96_v1,
  title = {optimal layer selection (intermediate layer utility)},
  author = {{Lacuna}},
  howpublished = {Lacuna research direction},
  year = {2026},
  url = {https://lacuna.tiptreesystems.com/direction/optimal-layer-selection-intermediate-layer-utility/rdr_c3ab5c9629114eaf9ead651030ee568f/v/1},
  note = {Version 1, frozen 2026-10-04. Accessed 2026-11-20},
}
About

These directions overlap with this one through shared papers.

See something wrong? Report a mistake