When we use a pre-trained deep neural network as a feature extractor, our default instinct is to harvest the embeddings from the very final layer. This design choice assumes that a network's final output represents the pinnacle of its predictive utility. However, a deep network is a pipeline of progressive abstraction. Early layers capture fine-grained spatial details, local textures, and structural boundaries. Middle layers assemble these into parts and shapes, while the final layers discard localized details to form abstract, invariant semantic concepts.
By relying exclusively on the final layer, we inadvertently throw away a massive repository of structural and textural information. A rapidly growing body of research demonstrates that hierarchical feature extraction—the practice of extracting and combining representations from multiple intermediate depths of a network—unlocks significant gains in robustness, sample efficiency, and performance across diverse downstream tasks.
The Intuition: Why the Final Layer is Not Enough
To understand why intermediate representations are valuable, consider the task of Image Quality Assessment (IQA). If an image of a landscape is compressed or corrupted by noise, its high-level semantic category ("landscape") remains perfectly intact. A classifier looking only at the final layer might see no difference between the pristine and corrupted images. However, the early layers of a convolutional neural network (CNN) are highly sensitive to the high-frequency changes caused by noise and compression artifacts.
To exploit this, Qin He et al. developed a no-reference quality assessment algorithm in Quality Assessment for Tone-Mapped HDR Images Using Multi-Scale and Multi-Layer Information that extracts features from multiple intermediate layers of a ResNet backbone. By combining low-level edge and texture features with deep semantic features, the model aligns closely with human subjective quality scores. Similarly, Vlad Hosu et al. demonstrated in Effective Aesthetics Prediction With Multi-Level Spatially Pooled Features that pooling activations across the entire depth of a pre-trained network allows a model to simultaneously evaluate fine-grained technical execution (sharpness, noise) and global composition.
This insight extends far beyond computer vision. In audio processing, Lipeng Dai et al. showed in Leveraging Multi-Level Features of ATST with Conformer-Based Dual-Branch Network for Sound Event Detection that splitting an audio transformer's internal layers into shallow and deep branches captures both local acoustic textures and abstract semantic meaning, dramatically improving sound event detection.
Mechanisms for Harvesting Multi-Layer Representations
Extracting features from every layer of a deep network introduces a practical challenge: high-dimensional data explosion. Researchers employ several distinct architectural mechanisms to fuse these multi-level features efficiently without overwhelming downstream models.
1. Stage-Wise Pooling and Linear Projections
Instead of extracting features from every single layer, architectures are often divided into macro-stages (e.g., residual blocks or transformer groups). Embeddings are harvested at the end of each stage. In Resource Aware Person Re-Identification Across Multiple Resolutions, Yan Wang et al. utilized global average pooling and linear projections at the end of each architectural stage. This creates a spectrum of embeddings that allows the system to dynamically trade off computation time for precision depending on image difficulty.
2. Attentive Fusion Connectors
Rather than statically concatenating layers, attention mechanisms can be used to dynamically weigh the importance of different depths based on the input context. Laure Ciernik et al. introduced an "attentive probe" in Beyond the final layer: Attentive multilayer fusion for vision transformers that uses cross-attention to dynamically fuse representations from all layers of a frozen Vision Transformer (ViT). This preserves mid-level structural cues that are typically discarded by the final layer, boosting downstream accuracy across dozens of datasets.
3. Multi-Exit Classifiers
For spatial localization tasks, intermediate classifiers can be attached directly to different depths of a backbone network. In AME-CAM: Attentive Multiple-Exit CAM for Weakly Supervised Segmentation on MRI Brain Tumor, Xinrong Hu attached intermediate classifiers to multiple stages of a network to extract class activation maps at varying resolutions. This allows the model to capture both fine-grained spatial boundaries in shallow layers and semantic tumor identities in deep layers using only image-level labels.
Input Image ---> [ Shallow Layers ] ---> Intermediate Features (Textures/Edges) ---> [ Attention / Fusion ]
| |
v v
[ Middle Layers ] ---> Intermediate Features (Parts/Shapes) ---> [ Downstream Task ]
| ^
v |
[ Deep Layers ] ---> Final Features (Abstract Semantics) ------------+
Why It Works: Generalization and Robustness
Beyond capturing fine details, hierarchical feature extraction fundamentally alters how a model generalizes to new, unseen domains.
When a deep network is trained on a specific dataset, its final layers become highly specialized to the training domain's label distribution and superficial biases (such as a specific background style or lighting condition). The earlier layers, however, learn fundamental geometric or structural invariants that remain stable across domains.
Domain Generalization
In Towards Domain Generalization for ECG and EEG Classification: Algorithms and Benchmarks, Aristotelis Ballas et al. demonstrated that integrating features from multiple layers of a deep network preserves core signal signatures that are less likely to change across different hospitals or patient populations. By accessing earlier layers, the architecture retains structural information that prevents the model from overfitting to domain-specific noise in the final layers.
Multi-Domain Feature Banks
Because different layers represent different levels of abstraction, a single pre-trained network can be treated as a diverse library of distinct feature spaces. Nikita Dvornik et al. leveraged this in Selecting Relevant Features from a Multi-domain Representation for Few-Shot Classification, showing that treating different layers of a single network as a multi-domain feature bank allows a model to select the most relevant intermediate representations for a new task, significantly outperforming models that rely solely on final-layer adaptation.
The "Free Lunch" for Multimodal Models
This paradigm has also proven highly effective for Multimodal Large Language Models (MLLMs). In Dense Connector for MLLMs, Huanjin Yao et al. demonstrated that feeding intermediate representations from a frozen vision encoder into an LLM acts as a computational "free lunch." Because these intermediate features are already computed during the standard forward pass, aggregating them provides the language model with rich spatial and fine-grained visual cues without adding any computational overhead to the vision stage.
By treating a deep neural network not as a monolithic vector generator, but as a rich, multi-layered hierarchy of representations, modern architectures can preserve the delicate balance between local detail and global abstraction.
Key Foundational Work
Exploiting Hierarchical Activations of Neural Network for Image Retrieval (2016) — contributed 1 concept elements to this direction.
Visual Tracking with Fully Convolutional Networks (2015) — contributed 1 concept elements to this direction.