Abstract: Transformer-based pre-trained language models have proven to be effective for learning contextualized language representation. However, current approaches only take advantage of the output of the encoder's final layer when fine-tuning the downstream tasks. We argue that only taking single layer's output restricts the power of pre-trained representation. Thus we deepen the representation learned by the model by fusing the hidden representation in terms of an explicit HIdden Representation Extractor (HIRE), which automatically absorbs the complementary representation with respect to the output from the final layer. Utilizing RoBERTa as the backbone encoder, our proposed improvement over the pre-trained models is shown effective on multiple natural language understanding tasks and help our model rival with the state-of-the-art models on the GLUE benchmark.
Introduction
Modern Natural Language Processing (NLP) relies heavily on deep Transformer-based models like BERT or RoBERTa. Standard practice involves pre-training a model on a massive corpus and then "fine-tuning" it for specific tasks. During fine-tuning, researchers typically only use the output of the encoder's final layer.
This paper argues that this "onefold use" of the last layer is a bottleneck. Previous research suggests that different layers of a Transformer capture different types of information: lower layers often handle surface-level features, while middle layers capture transferable syntactic and semantic features. By ignoring these intermediate "hidden" representations, we are essentially throwing away a wealth of contextual information that could improve model performance.
Methodology: The HIRE Framework
The authors propose a two-part architectural addition to the standard Transformer backbone (specifically RoBERTa-large):
1. HIdden Representation Extractor (HIRE)
Instead of just looking at the top, HIRE takes the hidden states from every layer (including the initial embedding). It uses a shared 2-layer Bidirectional GRU (Bi-GRU) to summarize each layer's state into a fixed-size vector. Through an Adaptive Weighting mechanism, it calculates an "importance score" for each layer using a specialized attention mechanism. This allows the model to dynamically decide which layers are most relevant for a specific input or task, resulting in a "complementary representation" that captures what the final layer might have overlooked.
2. Fusion Network
This module integrates the final layer's output with the complementary representation from HIRE. They use a sophisticated fusion approach, combining the two representations via concatenation, element-wise addition, and the Hadamard product (element-wise multiplication). A final 2-layer Bi-GRU processes this combined data to produce the final contextualized representation used for the task-specific output.
Experiments and Results
The model was evaluated on the GLUE (General Language Understanding Evaluation) benchmark, which includes nine diverse tasks ranging from sentiment analysis (SST-2) to natural language inference (MNLI, QNLI).
- Performance Boost: The RoBERTa+HIRE model outperformed the standard RoBERTa-large baseline on 5 out of 9 GLUE tasks.
- New SOTA: At the time of writing, the model achieved a state-of-the-art result on the SST-2 dataset with an accuracy of 97.1%.
- Significant Gains in Complexity: On the CoLA dataset (which tests grammatical "sense"), the model saw a significant jump of +1.7 points over the baseline, suggesting that hidden layers are particularly useful for complex linguistic judgments.
- Ablation Success: Removing either HIRE or the Fusion Network led to a drop in performance, proving that both the extraction of hidden data and the specific way it is fused are critical.
Discussion: Why Hidden Layers Matter
A key insight from the study is that the importance of specific layers isn't static. HIRE calculates importance scores per example and per task.
As shown in the heatmap above, different tasks have distinct "layer-importance signatures." For example, tasks like STS-B and RTE seem to require a more even contribution from all layers compared to sentiment analysis. The authors found that for many tasks, the most important information actually resides in the middle layers, confirming that the final layer often specializes too much on the pre-training objective (like language modeling) rather than the downstream task.
Conclusion and Future Work
HIRE provides a non-invasive way to improve pre-trained models without modifying their internal architecture or pre-training weights. However, this comes with a computational cost: adding HIRE and the Fusion Network increases the total parameter count from 355M to 437M. Future work could explore whether using more efficient "lite" Transformer layers for fusion would yield better results or improved inference speed.
Summary updated